[PATCHv2 0/2] Support template files for wic

[PATCHv2 0/2] Support template files for wic

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This adds support for .wks.in files as WKS_FILE.

This is a re-send of the RFC PATCH series as a regular PATCH series, to
attempt to get the feature merged. There's a clear value-add, there were no
major objections to the RFC series, and further improvements can be added on
from here.

These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another
templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.


The following changes since commit 646c366c2566bd8dd6f73681cea9f5b021589a56:

  gst-player: upgrade to latest HEAD (2016-06-27 14:08:37 +0100)

are available in the git repository at:

  git://github.com/kergoth/openembedded-core wks.in2
  https://github.com/kergoth/openembedded-core/tree/wks.in2

Christopher Larson (2):
  image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
  image_types.bbclass: support template .wks.in files for wic

 meta/classes/image_types.bbclass | 49 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)

-- 
2.8.0

[PATCHv2 1/2] image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This is a bit nicer to work with, and easier to override.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image_types.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 72e8ed3..f7de756 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -215,7 +215,8 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}"
-do_image_wic[file-checksums] += "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.8.0

[PATCHv2 2/2] image_types.bbclass: support template .wks.in files for wic

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another
templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image_types.bbclass | 46 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index f7de756..2b97397 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -218,6 +218,52 @@ USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 
+python do_write_wks_template () {
+    """Write out expanded template contents to WKS_FULL_PATH."""
+    import re
+
+    template_body = d.getVar('_WKS_TEMPLATE', True)
+
+    # Remove any remnant variable references left behind by the expansion
+    # due to undefined variables
+    expand_var_regexp = re.compile(r"\${[^{}@\n\t :]+}")
+    while True:
+        new_body = re.sub(expand_var_regexp, '', template_body)
+        if new_body == template_body:
+            break
+        else:
+            template_body = new_body
+
+    wks_file = d.getVar('WKS_FULL_PATH', True)
+    with open(wks_file, 'w') as f:
+        f.write(template_body)
+}
+
+python () {
+    if d.getVar('USING_WIC', True):
+        wks_file_u = d.getVar('WKS_FULL_PATH', False)
+        wks_file = d.expand(wks_file_u)
+        base, ext = os.path.splitext(wks_file)
+        if ext == '.in' and os.path.exists(wks_file):
+            wks_out_file = os.path.join(d.getVar('WORKDIR', True), os.path.basename(base))
+            d.setVar('WKS_FULL_PATH', wks_out_file)
+            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
+            d.setVar('WKS_FILE_CHECKSUM', '${WKS_TEMPLATE_PATH}:True')
+
+            try:
+                with open(wks_file, 'r') as f:
+                    body = f.read()
+            except (IOError, OSError) as exc:
+                pass
+            else:
+                # Previously, I used expandWithRefs to get the dependency list
+                # and add it to WICVARS, but there's no point re-parsing the
+                # file in process_wks_template as well, so just put it in
+                # a variable and let the metadata deal with the deps.
+                d.setVar('_WKS_TEMPLATE', body)
+                bb.build.addtask('do_write_wks_template', 'do_image_wic', None, d)
+}
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo
-- 
2.8.0

Re: [PATCHv2 0/2] Support template files for wic

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Fri, Jul 1, 2016 at 2:27 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> This adds support for .wks.in files as WKS_FILE.
>
> This is a re-send of the RFC PATCH series as a regular PATCH series, to
> attempt to get the feature merged. There's a clear value-add, there were no
> major objections to the RFC series, and further improvements can be added
> on
> from here.
>
> These files are treated as the contents of a bitbake variable, so usual
> bitbake variable references are supported. I considered using another
> templating mechanism, for example the one used by yocto-layer, but then
> we'd
> end up largely mapping metadata variables to template fields anyway, which
> is
> a pointless indirection. Let bitbake expand the variables directly instead.
>
> This feature lets us, for example, reference ${APPEND} in --append, and
> avoid
> hardcoding the serial console tty in the wks file, and let the user's
> changes
> to APPEND affect wic the way they do the other image construction
> mechanisms.
>
> The template is read in and set in a variable at parse time, so changes to
> the
> variables referenced by the template will result in rebuilding the image.
>

v2 changes: simplified by dropping the prefuncs bits.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2073 bytes --]

Re: [PATCHv2 0/2] Support template files for wic

[Ed Bartosh]

On Fri, Jul 01, 2016 at 02:27:00PM -0700, Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> This adds support for .wks.in files as WKS_FILE.
> 
> This is a re-send of the RFC PATCH series as a regular PATCH series, to
> attempt to get the feature merged. There's a clear value-add, there were no
> major objections to the RFC series, and further improvements can be added on
> from here.
> 
> These files are treated as the contents of a bitbake variable, so usual
> bitbake variable references are supported. I considered using another
> templating mechanism, for example the one used by yocto-layer, but then we'd
> end up largely mapping metadata variables to template fields anyway, which is
> a pointless indirection. Let bitbake expand the variables directly instead.
> 
> This feature lets us, for example, reference ${APPEND} in --append, and avoid
> hardcoding the serial console tty in the wks file, and let the user's changes
> to APPEND affect wic the way they do the other image construction mechanisms.
> 
> The template is read in and set in a variable at parse time, so changes to the
> variables referenced by the template will result in rebuilding the image.
> 
> 
> The following changes since commit 646c366c2566bd8dd6f73681cea9f5b021589a56:
> 
>   gst-player: upgrade to latest HEAD (2016-06-27 14:08:37 +0100)
> 
> are available in the git repository at:
> 
>   git://github.com/kergoth/openembedded-core wks.in2
>   https://github.com/kergoth/openembedded-core/tree/wks.in2
> 
> Christopher Larson (2):
>   image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
>   image_types.bbclass: support template .wks.in files for wic
> 
>  meta/classes/image_types.bbclass | 49 +++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 48 insertions(+), 1 deletion(-)
> 

Thank you for the patchset!

A bit unrelated note. You seem to know wic functionality in meta/classes/image* better
than anyone else. Would you mind adding this knowledge to wic documentation in development manual?

--
Regards,
Ed