[for-master][RFC PATCH 0/3] Support template files for wic wks

[for-master][RFC PATCH 0/3] Support template files for wic wks

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This adds support for .wks.in files as WKS_FILE.

These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another
templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.

It's been suggested that the name should better reflect how its contents are
used, i.e. .bbin or so. I've also considered including something like 'bbv' or
similar, to indicate its contents are that of a bitbake value. I think this
should be discussed.

I'd love to hear questions or comments on this. I think the value is clear,
and it's something we need to solve, but I'd like to hear if others agree or
disagree on that, and whether this implementation is appropriate.


The following changes since commit 1a0e56630c5c27d8899dd0979ae0b86bbe227881:

  utils.bbclass: note for deprecated base_contains (2016-04-29 07:53:58 +0100)

are available in the git repository at:

  git://github.com/kergoth/openembedded-core wks.in
  https://github.com/kergoth/openembedded-core/tree/wks.in

Christopher Larson (3):
  image.bbclass: append to prefuncs/postfuncs for do_image_*
  image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
  image_types.bbclass: support template .wks.in files for wic

 meta/classes/image.bbclass       |  4 ++--
 meta/classes/image_types.bbclass | 38 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 39 insertions(+), 3 deletions(-)

-- 
2.8.0

[for-master][RFC PATCH 1/3] image.bbclass: append to prefuncs/postfuncs for do_image_*

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

These are list style variables, so append to them rather than blowing away any
existing value.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 4542e95..bcb96aa 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -436,9 +436,9 @@ python () {
         d.setVar('do_image_%s' % t, '\n'.join(cmds))
         d.setVarFlag('do_image_%s' % t, 'func', '1')
         d.setVarFlag('do_image_%s' % t, 'fakeroot', '1')
-        d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size')
-        d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks')
         d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages))
+        d.appendVarFlag('do_image_%s' % t, 'prefuncs', ' ' + debug + 'set_image_size')
+        d.appendVarFlag('do_image_%s' % t, 'postfuncs', ' create_symlinks')
         d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps))
         d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
 
-- 
2.8.0

[for-master][RFC PATCH 2/3] image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This is a bit nicer to work with, and easier to override.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image_types.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 53af7ca..eed0373 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -208,7 +208,8 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}"
-do_image_wic[file-checksums] += "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 
 EXTRA_IMAGECMD = ""
 
-- 
2.8.0

[for-master][RFC PATCH 3/3] image_types.bbclass: support template .wks.in files for wic

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another
templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index eed0373..90b5a46 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -206,6 +206,16 @@ IMAGE_CMD_wic () {
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 
+python process_wks_template () {
+    """Write out expanded template contents to WKS_FULL_PATH."""
+    template_body = d.getVar('_WKS_TEMPLATE', True)
+    if template_body:
+        wks_file = d.getVar('WKS_FULL_PATH', True)
+        with open(wks_file, 'w') as f:
+            f.write(template_body)
+}
+do_image_wic[prefuncs] += 'process_wks_template'
+
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}"
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
@@ -298,3 +308,28 @@ IMAGE_TYPES_MASKED ?= ""
 # The WICVARS variable is used to define list of bitbake variables used in wic code
 # variables from this list is written to <image>.env file
 WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
+
+python () {
+    """Read in and set up wks file template for wic."""
+    if d.getVar('USING_WIC', True):
+        wks_file_u = d.getVar('WKS_FULL_PATH', False)
+        wks_file = d.expand(wks_file_u)
+        base, ext = os.path.splitext(wks_file)
+        if ext == '.in' and os.path.exists(wks_file):
+            wks_out_file = os.path.join(d.getVar('WORKDIR', True), os.path.basename(base))
+            d.setVar('WKS_FULL_PATH', wks_out_file)
+            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
+            d.setVar('WKS_FILE_CHECKSUM', '${WKS_TEMPLATE_PATH}:True')
+
+            try:
+                with open(wks_file, 'r') as f:
+                    body = f.read()
+            except (IOError, OSError) as exc:
+                pass
+            else:
+                # Previously, I used expandWithRefs to get the dependency list
+                # and add it to WICVARS, but there's no point re-parsing the
+                # file in process_wks_template as well, so just put it in
+                # a variable and let the metadata deal with the deps.
+                d.setVar('_WKS_TEMPLATE', body)
+}
-- 
2.8.0

Re: [for-master][RFC PATCH 0/3] Support template files for wic wks

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

On Mon, May 2, 2016 at 7:23 PM, Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> This adds support for .wks.in files as WKS_FILE.
>
> These files are treated as the contents of a bitbake variable, so usual
> bitbake variable references are supported. I considered using another
> templating mechanism, for example the one used by yocto-layer, but then
> we'd
> end up largely mapping metadata variables to template fields anyway, which
> is
> a pointless indirection. Let bitbake expand the variables directly instead.
>
> This feature lets us, for example, reference ${APPEND} in --append, and
> avoid
> hardcoding the serial console tty in the wks file, and let the user's
> changes
> to APPEND affect wic the way they do the other image construction
> mechanisms.
>
> The template is read in and set in a variable at parse time, so changes to
> the
> variables referenced by the template will result in rebuilding the image.
>
> It's been suggested that the name should better reflect how its contents
> are
> used, i.e. .bbin or so. I've also considered including something like
> 'bbv' or
> similar, to indicate its contents are that of a bitbake value. I think this
> should be discussed.
>
> I'd love to hear questions or comments on this. I think the value is clear,
> and it's something we need to solve, but I'd like to hear if others agree
> or
> disagree on that, and whether this implementation is appropriate.


One downside to this approach is, afaik,  that it's the first serious
deviation between wic outside and wic as an image type. One alternative
would be to parse the template at parse time to inject vars into WICVARS,
but let wic process the template, but then of course a change to the
template without re-running bitabke wouldn't update WICVARS and the written
env files.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2601 bytes --]

Re: [for-master][RFC PATCH 0/3] Support template files for wic wks

[Ed Bartosh]

On Mon, May 02, 2016 at 12:23:54PM -0700, Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
> 
> This adds support for .wks.in files as WKS_FILE.
> 
> These files are treated as the contents of a bitbake variable, so usual
> bitbake variable references are supported. I considered using another
> templating mechanism, for example the one used by yocto-layer, but then we'd
> end up largely mapping metadata variables to template fields anyway, which is
> a pointless indirection. Let bitbake expand the variables directly instead.
> 
> This feature lets us, for example, reference ${APPEND} in --append, and avoid
> hardcoding the serial console tty in the wks file, and let the user's changes
> to APPEND affect wic the way they do the other image construction mechanisms.
> 
> The template is read in and set in a variable at parse time, so changes to the
> variables referenced by the template will result in rebuilding the image.
> 
> It's been suggested that the name should better reflect how its contents are
> used, i.e. .bbin or so. I've also considered including something like 'bbv' or
> similar, to indicate its contents are that of a bitbake value. I think this
> should be discussed.
> 
> I'd love to hear questions or comments on this. I think the value is clear,
> and it's something we need to solve, but I'd like to hear if others agree or
> disagree on that, and whether this implementation is appropriate.
> 
The patchset looks good to me, but I'd also like to hear opinions of
people more experienced in image creation internals.

> 
> The following changes since commit 1a0e56630c5c27d8899dd0979ae0b86bbe227881:
> 
>   utils.bbclass: note for deprecated base_contains (2016-04-29 07:53:58 +0100)
> 
> are available in the git repository at:
> 
>   git://github.com/kergoth/openembedded-core wks.in
>   https://github.com/kergoth/openembedded-core/tree/wks.in
> 
> Christopher Larson (3):
>   image.bbclass: append to prefuncs/postfuncs for do_image_*
>   image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
>   image_types.bbclass: support template .wks.in files for wic
> 
>  meta/classes/image.bbclass       |  4 ++--
>  meta/classes/image_types.bbclass | 38 +++++++++++++++++++++++++++++++++++++-
>  2 files changed, 39 insertions(+), 3 deletions(-)
> 
> -- 
> 2.8.0
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
--
Regards,
Ed

Re: [for-master][RFC PATCH 0/3] Support template files for wic wks

[Jeremy ROSEN]

Le 02/05/2016 21:23, Christopher Larson a écrit :
> From: Christopher Larson <chris_larson@mentor.com>
> 
> This adds support for .wks.in files as WKS_FILE.
> 
> These files are treated as the contents of a bitbake variable, so 
> usual
> bitbake variable references are supported. I considered using another
> templating mechanism, for example the one used by yocto-layer, but 
> then we'd
> end up largely mapping metadata variables to template fields anyway, 
> which is
> a pointless indirection. Let bitbake expand the variables directly 
> instead.
> 
> This feature lets us, for example, reference ${APPEND} in --append, 
> and avoid
> hardcoding the serial console tty in the wks file, and let the user's 
> changes
> to APPEND affect wic the way they do the other image construction 
> mechanisms.
> 
> The template is read in and set in a variable at parse time, so 
> changes to the
> variables referenced by the template will result in rebuilding the 
> image.
> 
> It's been suggested that the name should better reflect how its 
> contents are
> used, i.e. .bbin or so. I've also considered including something like 
> 'bbv' or
> similar, to indicate its contents are that of a bitbake value. I think 
> this
> should be discussed.
> 
> I'd love to hear questions or comments on this. I think the value is 
> clear,
> and it's something we need to solve, but I'd like to hear if others 
> agree or
> disagree on that, and whether this implementation is appropriate.
> 
> 


We have been discussing this a bit with Chris on IRC and i'll add a 
couple of thoughts we had so that discussion isn't completely lost.

As a final user of yocto, I find the need for templating in wic very 
usefull, it would allow me to specify disk size in the machine 
specification and adapt the wic image accordingly, or use different disk 
names ( mmcblk vs sda)

however my need is more generic and I think it would be interesting to 
consider a bigger problem : using yocto variables in configuration 
files. (I mean files in /etc/ here, not yocto configuration)

It is a pretty common problem to have to tailor configuration files for 
particular machines, distro features, optional dependencies etc... this 
is possible to do using OVERRIDES but it can only be done at the 
whole-file level, not within a configuration file. Moreover it makes 
some complex cases very tricky. let's present an example

It is pretty common to need to include the build-machine's ip address 
in a configuration file. This can be usefull to send logs if the build 
machine is also the developer's machine. The ip address of the build 
machine can be easily obtained via python and put into the bitbake 
environement, but using that variable to set a value in a random file in 
/etc is pretty tricky.

similarly, I like to include my public ssh key as an authorized key for 
root on my builds, il makes testing way easier

I think having a generic templating mechanism that would recognise some 
source files as needing template extension would be awesome. I don't 
understand fully how bitbake/yocto works, so I can't design a complete 
archtecture, but here are a couple of thought we had with Chris on IRC

* recognise *.in in the same way *.patch are recognized. Is there a 
risk of improper expansion ? maybe specify that templating is needed as 
a parameter in SRC_URI
* should expansion be in do_fetch or do_patch or somewhere else... 
images tend to skip do_patch, but do_fetch is on the bitbake side IIUC. 
would that be a problem ?
* interaction with sstate and rebuilding needs to be considered... 
changes need to be detected post-templating obviously

Overall, I really like the idea of this patch, there are lot of things 
in a .wks that need expansion, but I think there is even more in every 
other configuration file of the system.

Hope my random thoughts helped

Regards

Jérémy Rosen

Re: [for-master][RFC PATCH 0/3] Support template files for wic wks

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 4973 bytes --]

On Sun, May 8, 2016 at 9:05 AM, Jeremy ROSEN <jeremy.rosen@smile.fr> wrote:

> Le 02/05/2016 21:23, Christopher Larson a écrit :
>
>> From: Christopher Larson <chris_larson@mentor.com>
>>
>> This adds support for .wks.in files as WKS_FILE.
>>
>> These files are treated as the contents of a bitbake variable, so usual
>> bitbake variable references are supported. I considered using another
>> templating mechanism, for example the one used by yocto-layer, but then
>> we'd
>> end up largely mapping metadata variables to template fields anyway,
>> which is
>> a pointless indirection. Let bitbake expand the variables directly
>> instead.
>>
>> This feature lets us, for example, reference ${APPEND} in --append, and
>> avoid
>> hardcoding the serial console tty in the wks file, and let the user's
>> changes
>> to APPEND affect wic the way they do the other image construction
>> mechanisms.
>>
>> The template is read in and set in a variable at parse time, so changes
>> to the
>> variables referenced by the template will result in rebuilding the image.
>>
>> It's been suggested that the name should better reflect how its contents
>> are
>> used, i.e. .bbin or so. I've also considered including something like
>> 'bbv' or
>> similar, to indicate its contents are that of a bitbake value. I think
>> this
>> should be discussed.
>>
>> I'd love to hear questions or comments on this. I think the value is
>> clear,
>> and it's something we need to solve, but I'd like to hear if others agree
>> or
>> disagree on that, and whether this implementation is appropriate.
>>
>>
>>
>
> We have been discussing this a bit with Chris on IRC and i'll add a couple
> of thoughts we had so that discussion isn't completely lost.
>
> As a final user of yocto, I find the need for templating in wic very
> usefull, it would allow me to specify disk size in the machine
> specification and adapt the wic image accordingly, or use different disk
> names ( mmcblk vs sda)
>
> however my need is more generic and I think it would be interesting to
> consider a bigger problem : using yocto variables in configuration files.
> (I mean files in /etc/ here, not yocto configuration)
>
> It is a pretty common problem to have to tailor configuration files for
> particular machines, distro features, optional dependencies etc... this is
> possible to do using OVERRIDES but it can only be done at the whole-file
> level, not within a configuration file. Moreover it makes some complex
> cases very tricky. let's present an example
>
> It is pretty common to need to include the build-machine's ip address in a
> configuration file. This can be usefull to send logs if the build machine
> is also the developer's machine. The ip address of the build machine can be
> easily obtained via python and put into the bitbake environement, but using
> that variable to set a value in a random file in /etc is pretty tricky.
>
> similarly, I like to include my public ssh key as an authorized key for
> root on my builds, il makes testing way easier
>
> I think having a generic templating mechanism that would recognise some
> source files as needing template extension would be awesome. I don't
> understand fully how bitbake/yocto works, so I can't design a complete
> archtecture, but here are a couple of thought we had with Chris on IRC
>
> * recognise *.in in the same way *.patch are recognized. Is there a risk
> of improper expansion ? maybe specify that templating is needed as a
> parameter in SRC_URI
> * should expansion be in do_fetch or do_patch or somewhere else... images
> tend to skip do_patch, but do_fetch is on the bitbake side IIUC. would that
> be a problem ?
> * interaction with sstate and rebuilding needs to be considered... changes
> need to be detected post-templating obviously
>
> Overall, I really like the idea of this patch, there are lot of things in
> a .wks that need expansion, but I think there is even more in every other
> configuration file of the system.
>
> Hope my random thoughts helped



This is an interesting concept, certainly a generic templating mechanism
for source files may be of use. I expect we could get this merged, and open
a new yocto feature request in bugzilla to add a more generic mechanism,
possibly based on this sort of thing.

The main concern I'd have is making sure changes to the referenced
variables, or changes to the list of referenced variables, changes
appropriate checksums, so as was the case in this patch series, they'd have
to be read in at parse time to gather that info. As long as the files are
relatively small, it should be reasonable to do so, and I expect the common
case will be small config files, so I doubt it'd be a blocking issue.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 5642 bytes --]

Re: [for-master][RFC PATCH 0/3] Support template files for wic wks

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2145 bytes --]

On Thu, May 12, 2016 at 11:06 AM, Christopher Larson <clarson@kergoth.com>
wrote:

>
> On Sun, May 8, 2016 at 9:05 AM, Jeremy ROSEN <jeremy.rosen@smile.fr>
> wrote:
>
>> Le 02/05/2016 21:23, Christopher Larson a écrit :
>>
>>> From: Christopher Larson <chris_larson@mentor.com>
>>>
>>> This adds support for .wks.in files as WKS_FILE.
>>>
>>> These files are treated as the contents of a bitbake variable, so usual
>>> bitbake variable references are supported. I considered using another
>>> templating mechanism, for example the one used by yocto-layer, but then
>>> we'd
>>> end up largely mapping metadata variables to template fields anyway,
>>> which is
>>> a pointless indirection. Let bitbake expand the variables directly
>>> instead.
>>>
>>> This feature lets us, for example, reference ${APPEND} in --append, and
>>> avoid
>>> hardcoding the serial console tty in the wks file, and let the user's
>>> changes
>>> to APPEND affect wic the way they do the other image construction
>>> mechanisms.
>>>
>>> The template is read in and set in a variable at parse time, so changes
>>> to the
>>> variables referenced by the template will result in rebuilding the image.
>>>
>>> It's been suggested that the name should better reflect how its contents
>>> are
>>> used, i.e. .bbin or so. I've also considered including something like
>>> 'bbv' or
>>> similar, to indicate its contents are that of a bitbake value. I think
>>> this
>>> should be discussed.
>>>
>>> I'd love to hear questions or comments on this. I think the value is
>>> clear,
>>> and it's something we need to solve, but I'd like to hear if others
>>> agree or
>>> disagree on that, and whether this implementation is appropriate.
>>
>>
Given that there don't seem to be any real objections here, I'm going to
re-submit this as a non-RFC patch series to try to get it merged. If anyone
has any issues adding this feature, let me know. Thanks for the feedback.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3045 bytes --]