[PATCH v2 0/2] devtool: create kernel config fragment

[PATCH v2 0/2] devtool: create kernel config fragment

[Markus Lehtonen]

Second iteration of rudimentary support for creating kernel config fragments
with devtool.  When doing 'update-recipe' devtool diffs .config.baseline and
.config (under ${S}) and creates/upates a config fragment which is added to
SRC_URI of the recipe. Unlike the in the first version of this patchset,
${S}/.config is now just a symlink to ${B}/.config. This way we only maintain
one copy of the config but it also means that the config is lost if ${B} is
cleaned.

[YOCTO #8999]

The following changes since commit 10351d2e7ab27d0f0311c491b70d0f0056a8f76b:

  pulseaudio: 6.0 -> 8.0 (2016-02-18 07:39:23 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/devtool/kernel

for you to fetch changes up to 08dcad8878e8715f29ff9b63c65f3e9c177117a4:

  devtool: update-recipe: create config fragment (2016-02-18 15:53:07 +0200)


Markus Lehtonen (2):
  devtool: sync: update kernel config
  devtool: update-recipe: create config fragment

 scripts/lib/devtool/standard.py | 62 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 60 insertions(+), 2 deletions(-)

-- 
2.6.2

[PATCH v2 1/2] devtool: sync: update kernel config

[Markus Lehtonen]

Copy kernel config is copied to the source directory at a later phase in
_extract_source() so that it gets copied when devtool sync is done, too.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index d12cc2e..a540a9b 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -577,12 +577,12 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d):
 
         bb.process.run('git tag -f devtool-patched', cwd=srcsubdir)
 
+        kconfig = None
         if bb.data.inherits_class('kernel-yocto', d):
             # Store generate and store kernel config
             logger.info('Generating kernel config')
             task_executor.exec_func('do_configure', False)
             kconfig = os.path.join(crd.getVar('B', True), '.config')
-            shutil.copy2(kconfig, srcsubdir)
 
 
         tempdir_localdir = os.path.join(tempdir, 'oe-local-files')
@@ -614,6 +614,10 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d):
 
             shutil.move(srcsubdir, srctree)
 
+        if kconfig:
+            logger.info('Copying kernel config to srctree')
+            shutil.copy2(kconfig, srctree)
+
     finally:
         bb.logger.setLevel(origlevel)
 
-- 
2.6.2

[PATCH v2 2/2] devtool: update-recipe: create config fragment

[Markus Lehtonen]

Create config fragment if the user makes modifications to kernel config.
User may change .config e.g. by directly editing it or by running the
'do_menuconfig' bitbake task. Devtool generates one monolithic fragment
by simply doing a diff between .config and .config.baseline files in the
source directory.  If either of these files is missing, the config
fragment is not gerenrated or updated. The output is a file,
'devtool-fragment.cfg' that gets added to SRC_URI in the recipe (as well
as copied into the 'oe-local-files' directory if that is present in the
source tree).

${S}/.config will be a symlink to ${B}/.config. We need to do this as
devtool is not able to access ${B} because ${B} is set in a .bbappend in
the workspace layer which is not parsed by devtool itself.

[YOCTO #8999]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 56 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 55 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index a540a9b..b763976 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -20,6 +20,7 @@ import os
 import sys
 import re
 import shutil
+import subprocess
 import tempfile
 import logging
 import argparse
@@ -775,6 +776,10 @@ def modify(args, config, basepath, workspace):
         if bb.data.inherits_class('kernel', rd):
             f.write('SRCTREECOVEREDTASKS = "do_validate_branches do_kernel_checkout '
                     'do_fetch do_unpack do_patch do_kernel_configme do_kernel_configcheck"\n')
+            f.write('\ndo_configure_append() {\n'
+                    '    cp ${B}/.config ${S}/.config.baseline\n'
+                    '    ln -sfT ${B}/.config ${S}/.config\n'
+                    '}\n')
         if initial_rev:
             f.write('\n# initial_rev: %s\n' % initial_rev)
             for commit in commits:
@@ -916,6 +921,33 @@ def _export_patches(srctree, rd, start_rev, destdir):
     return (updated, added, existing_patches)
 
 
+def _create_kconfig_diff(srctree, rd, outfile):
+    """Create a kconfig fragment"""
+    # Only update config fragment if both config files exist
+    orig_config = os.path.join(srctree, '.config.baseline')
+    new_config = os.path.join(srctree, '.config')
+    if os.path.exists(orig_config) and os.path.exists(new_config):
+        cmd = ['diff', '--new-line-format=%L', '--old-line-format=',
+               '--unchanged-line-format=', orig_config, new_config]
+        pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                stderr=subprocess.PIPE)
+        stdout, stderr = pipe.communicate()
+        if pipe.returncode == 1:
+            logger.info("Updating config fragment %s" % outfile)
+            with open(outfile, 'w') as fobj:
+                fobj.write(stdout)
+        elif pipe.returncode == 0:
+            logger.info("Would remove config fragment %s" % outfile)
+            if os.path.exists(outfile):
+                # Remove fragment file in case of empty diff
+                logger.info("Removing config fragment %s" % outfile)
+                os.unlink(outfile)
+        else:
+            raise bb.process.ExecutionError(cmd, pipe.returncode, stdout, stderr)
+        return True
+    return False
+
+
 def _export_local_files(srctree, rd, destdir):
     """Copy local files from srctree to given location.
        Returns three-tuple of dicts:
@@ -936,6 +968,7 @@ def _export_local_files(srctree, rd, destdir):
     updated = OrderedDict()
     added = OrderedDict()
     removed = OrderedDict()
+    local_files_dir = os.path.join(srctree, 'oe-local-files')
     git_files = _git_ls_tree(srctree)
     if 'oe-local-files' in git_files:
         # If tracked by Git, take the files from srctree HEAD. First get
@@ -946,11 +979,32 @@ def _export_local_files(srctree, rd, destdir):
                         env=dict(os.environ, GIT_WORK_TREE=destdir,
                                  GIT_INDEX_FILE=tmp_index))
         new_set = _git_ls_tree(srctree, tree, True).keys()
-    elif os.path.isdir(os.path.join(srctree, 'oe-local-files')):
+    elif os.path.isdir(local_files_dir):
         # If not tracked by Git, just copy from working copy
         new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
         bb.process.run(['cp', '-ax',
                         os.path.join(srctree, 'oe-local-files', '.'), destdir])
+    else:
+        new_set = []
+
+    # Special handling for kernel config
+    if bb.data.inherits_class('kernel-yocto', rd):
+        fragment_fn = 'devtool-fragment.cfg'
+        fragment_path = os.path.join(destdir, fragment_fn)
+        if _create_kconfig_diff(srctree, rd, fragment_path):
+            if os.path.exists(fragment_path):
+                if fragment_fn not in new_set:
+                    new_set.append(fragment_fn)
+                # Copy fragment to local-files
+                if os.path.isdir(local_files_dir):
+                    shutil.copy2(fragment_path, local_files_dir)
+            else:
+                if fragment_fn in new_set:
+                    new_set.remove(fragment_fn)
+                # Remove fragment from local-files
+                if os.path.exists(os.path.join(local_files_dir, fragment_fn)):
+                    os.unlink(os.path.join(local_files_dir, fragment_fn))
+
     if new_set is not None:
         for fname in new_set:
             if fname in existing_files:
-- 
2.6.2

Re: [PATCH v2 2/2] devtool: update-recipe: create config fragment

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Thu, Feb 18, 2016 at 7:19 AM Markus Lehtonen <
markus.lehtonen@linux.intel.com> wrote:

> Create config fragment if the user makes modifications to kernel config.
> User may change .config e.g. by directly editing it or by running the
> 'do_menuconfig' bitbake task. Devtool generates one monolithic fragment
> by simply doing a diff between .config and .config.baseline files in the
> source directory.  If either of these files is missing, the config
> fragment is not gerenrated or updated. The output is a file,
> 'devtool-fragment.cfg' that gets added to SRC_URI in the recipe (as well
> as copied into the 'oe-local-files' directory if that is present in the
> source tree).
>
> ${S}/.config will be a symlink to ${B}/.config. We need to do this as
> devtool is not able to access ${B} because ${B} is set in a .bbappend in
> the workspace layer which is not parsed by devtool itself.
>
> [YOCTO #8999]
>
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
>

If the devtool-fragment.cfg already exists, what's the behavior? Will it
update it with both the old and new changes, or will it create a second
fragment?

[-- Attachment #2: Type: text/html, Size: 1575 bytes --]

Re: [PATCH v2 2/2] devtool: update-recipe: create config fragment

[Markus Lehtonen]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]

On Thu, 2016-02-18 at 15:19 +0000, Christopher Larson wrote:
> 
> 
> On Thu, Feb 18, 2016 at 7:19 AM Markus Lehtonen <
> markus.lehtonen@linux.intel.com> wrote:
> > Create config fragment if the user makes modifications to kernel
> > config.
> > User may change .config e.g. by directly editing it or by running
> > the
> > 'do_menuconfig' bitbake task. Devtool generates one monolithic
> > fragment
> > by simply doing a diff between .config and .config.baseline files
> > in the
> > source directory.  If either of these files is missing, the config
> > fragment is not gerenrated or updated. The output is a file,
> > 'devtool-fragment.cfg' that gets added to SRC_URI in the recipe (as
> > well
> > as copied into the 'oe-local-files' directory if that is present in
> > the
> > source tree).
> > 
> > ${S}/.config will be a symlink to ${B}/.config. We need to do this
> > as
> > devtool is not able to access ${B} because ${B} is set in a
> > .bbappend in
> > the workspace layer which is not parsed by devtool itself.
> > 
> > [YOCTO #8999]
> > 
> > Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> If the devtool-fragment.cfg already exists, what's the behavior? Will
> it update it with both the old and new changes, or will it create a
> second fragment?
Currently, it will update the existing fragment.
   - Markus

[-- Attachment #2: Type: text/html, Size: 1743 bytes --]