[PATCH 0/1] bblayers sanity: bblayers.conf.sample specific update code

[PATCH 0/1] bblayers sanity: bblayers.conf.sample specific update code

[Patrick Ohly]

In an experimental branch of refkit I bumped LCONF_VERSION from 4 to 5
and ran into the problem mentioned in the commit message: instead of
raising an error as before, the code (incorrectly) claimed to have
updated the bblayers.conf file.

Perhaps refkit should have started with LCONF_VERSION as in the
current OE-core bblayers.conf.sample file to avoid being affected by
this - too late for that now, though.

As an ugly workaround I'm currently jumping from 4 to 8 directly,
which produces some extra (and wrong) output about "automatically
updated" before finally raising the expected error (twice, for some
reasons).

I'm not sure how this code was meant to work for different distros
based on OE-core. Depending on when a distro forked from OE-core and
how it continued with numbering its bblayers.conf.sample file, there
must have been such conflicts already before.

Note that I have only checked that the code gets called when placed in
the expected location, and the code itself was copied verbatim. As it
refers to ancient changes, I've not actually tried to set up a build
dir where that upgrade path is necessary. Perhaps the code could even
be removed entirely?

Patrick Ohly (1):
  bblayers sanity: bblayers.conf.sample specific update code

 meta/classes/sanity.bbclass               | 121 ++++-------------------
 meta/conf/lib/bblayers_update/__init__.py | 104 ++++++++++++++++++++-
 2 files changed, 130 insertions(+), 95 deletions(-)
 create mode 100644 meta/conf/lib/bblayers_update/__init__.py

base-commit: ccf630e78aad488da7b80f2981037d3d0559cfad
-- 
git-series 0.9.1

[PATCH 1/1] bblayers sanity: bblayers.conf.sample specific update code

[Patrick Ohly]

The code in sanity.bbclass was executed for all distros, regardless
whether they use the same numbering of bblayers.conf.sample as
OE-core.

This is problematic for distros which started at LCONF_VERSION=1: once
they reach the 5 to 7 (inclusive) number range, the code from
sanity.bbclass kicks in and automatically updates the version to that
number without actually changing anything else in bblayers.conf, which
leaves the developer out-of-sync with bblayers.conf.sample.

Now the code is in a Python module that gets placed alongside the
bblayers.conf.sample to which it belongs. Besides avoiding the problem
above, this also allows distros to write their own update code.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/sanity.bbclass               | 121 ++++-------------------
 meta/conf/lib/bblayers_update/__init__.py | 104 ++++++++++++++++++++-
 2 files changed, 130 insertions(+), 95 deletions(-)
 create mode 100644 meta/conf/lib/bblayers_update/__init__.py

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index e8064ac..5cc0d14 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -5,26 +5,6 @@
 SANITY_REQUIRED_UTILITIES ?= "patch diffstat makeinfo git bzip2 tar \
     gzip gawk chrpath wget cpio perl file which"
 
-def bblayers_conf_file(d):
-    return os.path.join(d.getVar('TOPDIR'), 'conf/bblayers.conf')
-
-def sanity_conf_read(fn):
-    with open(fn, 'r') as f:
-        lines = f.readlines()
-    return lines
-
-def sanity_conf_find_line(pattern, lines):
-    import re
-    return next(((index, line)
-        for index, line in enumerate(lines)
-        if re.search(pattern, line)), (None, None))
-
-def sanity_conf_update(fn, lines, version_var_name, new_version):
-    index, line = sanity_conf_find_line(r"^%s" % version_var_name, lines)
-    lines[index] = '%s = "%d"\n' % (version_var_name, new_version)
-    with open(fn, "w") as f:
-        f.write(''.join(lines))
-
 # Functions added to this variable MUST throw a NotImplementedError exception unless 
 # they successfully changed the config version in the config file. Exceptions
 # are used since exec_func doesn't handle return values.
@@ -76,7 +56,12 @@ is a good way to visualise the changes."""
     raise NotImplementedError(failmsg)
 }
 
+# Distros may have their own bblayers.conf.sample and corresponding
+# update code. The update code must be provided in a sanity_update_bblayers()
+# function in a "lib/bblayers-update" module which must be
+# in the same directory as the bblayers.conf.sample.
 SANITY_BBLAYERCONF_SAMPLE ?= "${COREBASE}/meta*/conf/bblayers.conf.sample"
+
 python oecore_update_bblayers() {
     # bblayers.conf is out of date, so see if we can resolve that
 
@@ -90,83 +75,29 @@ Please compare your file against bblayers.conf.sample and merge any changes befo
 is a good way to visualise the changes."""
     failmsg = d.expand(failmsg)
 
-    if not current_lconf:
+    import glob
+    samples = glob.glob(d.getVar('SANITY_BBLAYERCONF_SAMPLE'))
+    if len(samples) != 1:
+        # Ambiguous bblayers.conf.sample, must be dealt with manually.
+        # Should be avoided by overriding SANITY_BBLAYERCONF_SAMPLE.
         raise NotImplementedError(failmsg)
 
-    lines = []
-
-    if current_lconf < 4:
-        raise NotImplementedError(failmsg)
-
-    bblayers_fn = bblayers_conf_file(d)
-    lines = sanity_conf_read(bblayers_fn)
-
-    if current_lconf == 4 and lconf_version > 4:
-        topdir_var = '$' + '{TOPDIR}'
-        index, bbpath_line = sanity_conf_find_line('BBPATH', lines)
-        if bbpath_line:
-            start = bbpath_line.find('"')
-            if start != -1 and (len(bbpath_line) != (start + 1)):
-                if bbpath_line[start + 1] == '"':
-                    lines[index] = (bbpath_line[:start + 1] +
-                                    topdir_var + bbpath_line[start + 1:])
-                else:
-                    if not topdir_var in bbpath_line:
-                        lines[index] = (bbpath_line[:start + 1] +
-                                    topdir_var + ':' + bbpath_line[start + 1:])
-            else:
-                raise NotImplementedError(failmsg)
-        else:
-            index, bbfiles_line = sanity_conf_find_line('BBFILES', lines)
-            if bbfiles_line:
-                lines.insert(index, 'BBPATH = "' + topdir_var + '"\n')
-            else:
-                raise NotImplementedError(failmsg)
-
-        current_lconf += 1
-        sanity_conf_update(bblayers_fn, lines, 'LCONF_VERSION', current_lconf)
-        bb.note("Your conf/bblayers.conf has been automatically updated.")
-        return
-
-    elif current_lconf == 5 and lconf_version > 5:
-        # Null update, to avoid issues with people switching between poky and other distros
-        current_lconf = 6
-        sanity_conf_update(bblayers_fn, lines, 'LCONF_VERSION', current_lconf)
-        bb.note("Your conf/bblayers.conf has been automatically updated.")
-        return
-
-        status.addresult()
-
-    elif current_lconf == 6 and lconf_version > 6:
-        # Handle rename of meta-yocto -> meta-poky
-        # This marks the start of separate version numbers but code is needed in OE-Core
-        # for the migration, one last time.
-        layers = d.getVar('BBLAYERS').split()
-        layers = [ os.path.basename(path) for path in layers ]
-        if 'meta-yocto' in layers:
-            found = False
-            while True:
-                index, meta_yocto_line = sanity_conf_find_line(r'.*meta-yocto[\'"\s\n]', lines)
-                if meta_yocto_line:
-                    lines[index] = meta_yocto_line.replace('meta-yocto', 'meta-poky')
-                    found = True
-                else:
-                    break
-            if not found:
-                raise NotImplementedError(failmsg)
-            index, meta_yocto_line = sanity_conf_find_line('LCONF_VERSION.*\n', lines)
-            if meta_yocto_line:
-                lines[index] = 'POKY_BBLAYERS_CONF_VERSION = "1"\n'
-            else:
-                raise NotImplementedError(failmsg)
-            with open(bblayers_fn, "w") as f:
-                f.write(''.join(lines))
-            bb.note("Your conf/bblayers.conf has been automatically updated.")
-            return
-        current_lconf += 1
-        sanity_conf_update(bblayers_fn, lines, 'LCONF_VERSION', current_lconf)
-        bb.note("Your conf/bblayers.conf has been automatically updated.")
-        return
+    # Is there matching update code for the bblayers.conf.sample?
+    libdir = os.path.join(os.path.dirname(samples[0]), 'lib')
+    moduledir = os.path.join(libdir, 'bblayers_update')
+    if os.path.exists(moduledir):
+        old_path = sys.path.copy()
+        try:
+            sys.path.insert(0, libdir)
+            try:
+                from bblayers_update import sanity_update_bblayers
+            except:
+                bb.warn('Importing sanity_update_bblayers from %s failed.' % moduledir)
+                raise
+            if sanity_update_bblayers(d, failmsg):
+                return True
+        finally:
+            sys.path = old_path
 
     raise NotImplementedError(failmsg)
 }
diff --git a/meta/conf/lib/bblayers_update/__init__.py b/meta/conf/lib/bblayers_update/__init__.py
new file mode 100644
index 0000000..bd6693c
--- /dev/null
+++ b/meta/conf/lib/bblayers_update/__init__.py
@@ -0,0 +1,104 @@
+import bb
+
+def bblayers_conf_file(d):
+    return os.path.join(d.getVar('TOPDIR'), 'conf/bblayers.conf')
+
+def sanity_conf_read(fn):
+    with open(fn, 'r') as f:
+        lines = f.readlines()
+    return lines
+
+def sanity_conf_find_line(pattern, lines):
+    import re
+    return next(((index, line)
+        for index, line in enumerate(lines)
+        if re.search(pattern, line)), (None, None))
+
+def sanity_conf_update(fn, lines, version_var_name, new_version):
+    index, line = sanity_conf_find_line(r"^%s" % version_var_name, lines)
+    lines[index] = '%s = "%d"\n' % (version_var_name, new_version)
+    with open(fn, "w") as f:
+        f.write(''.join(lines))
+
+def sanity_update_bblayers(d, failmsg):
+    current_lconf = int(d.getVar('LCONF_VERSION'))
+    lconf_version = int(d.getVar('LAYER_CONF_VERSION'))
+
+    if not current_lconf:
+        return False
+
+    lines = []
+
+    if current_lconf < 4:
+        return False
+
+    bblayers_fn = bblayers_conf_file(d)
+    lines = sanity_conf_read(bblayers_fn)
+
+    if current_lconf == 4 and lconf_version > 4:
+        topdir_var = '$' + '{TOPDIR}'
+        index, bbpath_line = sanity_conf_find_line('BBPATH', lines)
+        if bbpath_line:
+            start = bbpath_line.find('"')
+            if start != -1 and (len(bbpath_line) != (start + 1)):
+                if bbpath_line[start + 1] == '"':
+                    lines[index] = (bbpath_line[:start + 1] +
+                                    topdir_var + bbpath_line[start + 1:])
+                else:
+                    if not topdir_var in bbpath_line:
+                        lines[index] = (bbpath_line[:start + 1] +
+                                    topdir_var + ':' + bbpath_line[start + 1:])
+            else:
+                return False
+        else:
+            index, bbfiles_line = sanity_conf_find_line('BBFILES', lines)
+            if bbfiles_line:
+                lines.insert(index, 'BBPATH = "' + topdir_var + '"\n')
+            else:
+                return False
+
+        current_lconf += 1
+        sanity_conf_update(bblayers_fn, lines, 'LCONF_VERSION', current_lconf)
+        bb.note("Your conf/bblayers.conf has been automatically updated.")
+        return True
+
+    elif current_lconf == 5 and lconf_version > 5:
+        # Null update, to avoid issues with people switching between poky and other distros
+        current_lconf = 6
+        sanity_conf_update(bblayers_fn, lines, 'LCONF_VERSION', current_lconf)
+        bb.note("Your conf/bblayers.conf has been automatically updated.")
+        return True
+
+        status.addresult()
+
+    elif current_lconf == 6 and lconf_version > 6:
+        # Handle rename of meta-yocto -> meta-poky
+        # This marks the start of separate version numbers but code is needed in OE-Core
+        # for the migration, one last time.
+        layers = d.getVar('BBLAYERS').split()
+        layers = [ os.path.basename(path) for path in layers ]
+        if 'meta-yocto' in layers:
+            found = False
+            while True:
+                index, meta_yocto_line = sanity_conf_find_line(r'.*meta-yocto[\'"\s\n]', lines)
+                if meta_yocto_line:
+                    lines[index] = meta_yocto_line.replace('meta-yocto', 'meta-poky')
+                    found = True
+                else:
+                    break
+            if not found:
+                return False
+            index, meta_yocto_line = sanity_conf_find_line('LCONF_VERSION.*\n', lines)
+            if meta_yocto_line:
+                lines[index] = 'POKY_BBLAYERS_CONF_VERSION = "1"\n'
+            else:
+                return False
+            with open(bblayers_fn, "w") as f:
+                f.write(''.join(lines))
+            bb.note("Your conf/bblayers.conf has been automatically updated.")
+            return True
+        current_lconf += 1
+        sanity_conf_update(bblayers_fn, lines, 'LCONF_VERSION', current_lconf)
+        d.setVar('LCONF_VERSION', current_lconf)
+        bb.note("Your conf/bblayers.conf has been automatically updated.")
+        return True
-- 
git-series 0.9.1

Re: [PATCH 0/1] bblayers sanity: bblayers.conf.sample specific update code

[Patrick Ohly]

Ping?

On Mon, 2017-05-29 at 16:49 +0200, Patrick Ohly wrote:
> In an experimental branch of refkit I bumped LCONF_VERSION from 4 to 5
> and ran into the problem mentioned in the commit message: instead of
> raising an error as before, the code (incorrectly) claimed to have
> updated the bblayers.conf file.
> 
> Perhaps refkit should have started with LCONF_VERSION as in the
> current OE-core bblayers.conf.sample file to avoid being affected by
> this - too late for that now, though.
> 
> As an ugly workaround I'm currently jumping from 4 to 8 directly,
> which produces some extra (and wrong) output about "automatically
> updated" before finally raising the expected error (twice, for some
> reasons).
> 
> I'm not sure how this code was meant to work for different distros
> based on OE-core. Depending on when a distro forked from OE-core and
> how it continued with numbering its bblayers.conf.sample file, there
> must have been such conflicts already before.
> 
> Note that I have only checked that the code gets called when placed in
> the expected location, and the code itself was copied verbatim. As it
> refers to ancient changes, I've not actually tried to set up a build
> dir where that upgrade path is necessary. Perhaps the code could even
> be removed entirely?
> 
> Patrick Ohly (1):
>   bblayers sanity: bblayers.conf.sample specific update code
> 
>  meta/classes/sanity.bbclass               | 121 ++++-------------------
>  meta/conf/lib/bblayers_update/__init__.py | 104 ++++++++++++++++++++-
>  2 files changed, 130 insertions(+), 95 deletions(-)
>  create mode 100644 meta/conf/lib/bblayers_update/__init__.py
> 
> base-commit: ccf630e78aad488da7b80f2981037d3d0559cfad