[PATCH 0/3] wic: glob support & minor fixes

[PATCH 0/3] wic: glob support & minor fixes

[Maciej Borzecki]

This patch series adds changes that enable globbing in IMAGE_BOOT_FILES.
Effectivey this makes use of with with Raspberry Pi a tad easier, when a
lengthy list of boot files can be replaced with just one entry. A related
change to meta-raspberrypi will be sent to the Yocto mailing list.

Maciej Borzecki (3):
  wic: IMAGE_BOOT_FILES format checks in bootimg-partition source
  wic: add globbing support in IMAGE_BOOT_FILES entries
  ref-manual: update IMAGE_BOOT_FILES entry

 documentation/ref-manual/ref-variables.xml         | 37 +++++++++++++++----
 .../lib/wic/plugins/source/bootimg-partition.py    | 43 +++++++++++++++++-----
 2 files changed, 64 insertions(+), 16 deletions(-)

-- 
1.9.3

[PATCH 1/3] wic: IMAGE_BOOT_FILES format checks in bootimg-partition source

[Maciej Borzecki]

Check for malformed entries in IMAGE_BOOT_FILES, fail early if such
entries were found.

Signed-off-by: Maciej Borzecki <maciej.borzecki@open-rnd.pl>
Signed-off-by: Maciek Borzecki <maciek.borzecki@gmail.com>
---
 scripts/lib/wic/plugins/source/bootimg-partition.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py b/scripts/lib/wic/plugins/source/bootimg-partition.py
index abf2494..564118a 100644
--- a/scripts/lib/wic/plugins/source/bootimg-partition.py
+++ b/scripts/lib/wic/plugins/source/bootimg-partition.py
@@ -90,6 +90,8 @@ class BootimgPartitionPlugin(SourcePlugin):
         for src_entry in re.findall(r'[\w;\-\./]+', boot_files):
             if ';' in src_entry:
                 dst_entry = tuple(src_entry.split(';'))
+                if not dst_entry[0] or not dst_entry[1]:
+                    msger.error('Malformed boot file entry: %s' % (src_entry))
             else:
                 dst_entry = (src_entry, src_entry)
 
-- 
1.9.3

[PATCH 2/3] wic: add globbing support in IMAGE_BOOT_FILES entries

[Maciej Borzecki]

Adding glob support for entries in IMAGE_BOOT_FILES. Files picked up by
glob are by default installed under their basename, as this is likely
most common use case. Target name for globbed entries specifies the
name of directory in which files will be installed withing the partition.

Signed-off-by: Maciej Borzecki <maciej.borzecki@open-rnd.pl>
Signed-off-by: Maciek Borzecki <maciek.borzecki@gmail.com>
---
 .../lib/wic/plugins/source/bootimg-partition.py    | 41 +++++++++++++++++-----
 1 file changed, 32 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py b/scripts/lib/wic/plugins/source/bootimg-partition.py
index 564118a..6ba39a0 100644
--- a/scripts/lib/wic/plugins/source/bootimg-partition.py
+++ b/scripts/lib/wic/plugins/source/bootimg-partition.py
@@ -29,6 +29,7 @@ import re
 from wic import msger
 from wic.pluginbase import SourcePlugin
 from wic.utils.oe.misc import *
+from glob import glob
 
 class BootimgPartitionPlugin(SourcePlugin):
     name = 'bootimg-partition'
@@ -87,7 +88,7 @@ class BootimgPartitionPlugin(SourcePlugin):
 
         # list of tuples (src_name, dst_name)
         deploy_files = []
-        for src_entry in re.findall(r'[\w;\-\./]+', boot_files):
+        for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files):
             if ';' in src_entry:
                 dst_entry = tuple(src_entry.split(';'))
                 if not dst_entry[0] or not dst_entry[1]:
@@ -100,14 +101,36 @@ class BootimgPartitionPlugin(SourcePlugin):
 
         for deploy_entry in deploy_files:
             src, dst = deploy_entry
-            src_path = os.path.join(bootimg_dir, src)
-            dst_path = os.path.join(hdddir, dst)
-
-            msger.debug('Install %s as %s' % (os.path.basename(src_path),
-                                              dst_path))
-            install_cmd = "install -m 0644 -D %s %s" \
-                          % (src_path, dst_path)
-            exec_cmd(install_cmd)
+            install_task = []
+            if '*' in src:
+                # by default install files under their basename
+                entry_name_fn = os.path.basename
+                if dst != src:
+                    # unless a target name was given, then treat name
+                    # as a directory and append a basename
+                    entry_name_fn = lambda name: \
+                                    os.path.join(dst,
+                                                 os.path.basename(name))
+
+                srcs = glob(os.path.join(bootimg_dir, src))
+
+                msger.debug('Globbed sources: %s' % (', '.join(srcs)))
+                for entry in srcs:
+                    entry_dst_name = entry_name_fn(entry)
+                    install_task.append((entry,
+                                         os.path.join(hdddir,
+                                                      entry_dst_name)))
+            else:
+                install_task = [(os.path.join(bootimg_dir, src),
+                                 os.path.join(hdddir, dst))]
+
+            for task in install_task:
+                src_path, dst_path = task
+                msger.debug('Install %s as %s' % (os.path.basename(src_path),
+                                                  dst_path))
+                install_cmd = "install -m 0644 -D %s %s" \
+                              % (src_path, dst_path)
+                exec_cmd(install_cmd)
 
         msger.debug('Prepare boot partition using rootfs in %s' % (hdddir))
         part.prepare_rootfs(cr_workdir, oe_builddir, hdddir,
-- 
1.9.3

[PATCH 3/3] ref-manual: update IMAGE_BOOT_FILES entry

[Maciej Borzecki]

Update entry for IMAGE_BOOT_FILES with information on glob patterns.

Signed-off-by: Maciej Borzecki <maciej.borzecki@open-rnd.pl>
Signed-off-by: Maciek Borzecki <maciek.borzecki@gmail.com>
---
 documentation/ref-manual/ref-variables.xml | 37 ++++++++++++++++++++++++------
 1 file changed, 30 insertions(+), 7 deletions(-)

diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index 0bae35a..bd857ef 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -3863,19 +3863,42 @@
             <glossdef>
                 <para>
                     A space-separated list of files installed into the
-                    boot partition when preparing an image.
-                    By default, the files are installed under the same name as
-                    the source files.
+                    boot partition when preparing an image using
+                    <filename>wic</filename> tool with
+                    <filename>bootimg-partition</filename> source
+                    plugin. By default, the files are installed under
+                    the same name as the source files.
+
                     To change the installed name, separate it from the
-                    original name with a semi-colon (;).
-                    Source files need to be located in
-                    <link linkend='var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></link>.
+                    original name with a semi-colon (;).  Source files
+                    need to be located in <link
+                    linkend='var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></link>.
                     Here are two examples:
 
                     <literallayout class="monospaced">
      IMAGE_BOOT_FILES = "u-boot.img uImage;kernel"
      IMAGE_BOOT_FILES = "u-boot.${UBOOT_SUFFIX} ${KERNEL_IMAGETYPE}"
-                    </literallayout></para>
+                    </literallayout>
+
+                    Alternatively source files can be picked up using
+                    a glob pattern. In this case the destination file
+                    will have the same as base name of the source file
+                    path. To install files into a directory within the
+                    target location pass its name after a semi-colon
+                    (;). Here are two examples:
+
+                    <literallayout class="monospaced">
+     IMAGE_BOOT_FILES = "bcm2835-bootfiles/*"
+     IMAGE_BOOT_FILES = "bcm2835-bootfiles/*;boot/"
+                    </literallayout>
+
+                    The first example installs all files from
+                    <filename>${DEPLOY_DIR_IMAGE}/bcm2835-bootfiles</filename>
+                    into the root of the target partition. The second
+                    example installs the same files into a
+                    <filename>boot</filename> directory within the
+                    target partition.
+                </para>
             </glossdef>
         </glossentry>
 
-- 
1.9.3

Re: [PATCH 0/3] wic: glob support & minor fixes

[Tom Zanussi]

On Wed, 2014-12-10 at 12:45 +0100, Maciej Borzecki wrote:
> This patch series adds changes that enable globbing in IMAGE_BOOT_FILES.
> Effectivey this makes use of with with Raspberry Pi a tad easier, when a
> lengthy list of boot files can be replaced with just one entry. A related
> change to meta-raspberrypi will be sent to the Yocto mailing list.
> 

Looks ok to me, and I tried it out with your raspberrypi layer changes
which generated the image (I didn't try any other image types to check
against regressions there, though nothing in here looks like it would
touch anything related to the other source plugins, so I'm not worried -
in the next milestone I hope to have some automated wic testing going to
make checking easy).

I do think, though, that we need to have the documentation for this in
the tool itself.  As it stands, the IMAGE_BOOT_FILES variable is
documented in the manual, but even then, there's no context that ties it
to the bootimg-partition plugin.  I think we need to add a mechanism so
that help text can be added to each plugin, which would be automatically
picked up and merged probably into a plugin-specific 'help plugins'
section in the tool.

The other thing I noticed, which has nothing to do with this patchset,
but along the same lines, is that the plugin machinery does find the
sdimage-raspberrypi.wks file to generate the image, as expected, but it
doesn't show up in 'wic list source-plugins', as you'd expect.

Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>

> Maciej Borzecki (3):
>   wic: IMAGE_BOOT_FILES format checks in bootimg-partition source
>   wic: add globbing support in IMAGE_BOOT_FILES entries
>   ref-manual: update IMAGE_BOOT_FILES entry
> 
>  documentation/ref-manual/ref-variables.xml         | 37 +++++++++++++++----
>  .../lib/wic/plugins/source/bootimg-partition.py    | 43 +++++++++++++++++-----
>  2 files changed, 64 insertions(+), 16 deletions(-)
> 
> -- 
> 1.9.3
> 

Re: [PATCH 0/3] wic: glob support & minor fixes

[Maciej Borzecki]

On 12/15 11:04, Tom Zanussi wrote:
> On Wed, 2014-12-10 at 12:45 +0100, Maciej Borzecki wrote:
> > This patch series adds changes that enable globbing in IMAGE_BOOT_FILES.
> > Effectivey this makes use of with with Raspberry Pi a tad easier, when a
> > lengthy list of boot files can be replaced with just one entry. A related
> > change to meta-raspberrypi will be sent to the Yocto mailing list.
> >
>
> Looks ok to me, and I tried it out with your raspberrypi layer changes
> which generated the image (I didn't try any other image types to check
> against regressions there, though nothing in here looks like it would
> touch anything related to the other source plugins, so I'm not worried -
> in the next milestone I hope to have some automated wic testing going to
> make checking easy).
>
> I do think, though, that we need to have the documentation for this in
> the tool itself.  As it stands, the IMAGE_BOOT_FILES variable is
> documented in the manual, but even then, there's no context that ties it
> to the bootimg-partition plugin.  I think we need to add a mechanism so
> that help text can be added to each plugin, which would be automatically
> picked up and merged probably into a plugin-specific 'help plugins'
> section in the tool.
>
> The other thing I noticed, which has nothing to do with this patchset,
> but along the same lines, is that the plugin machinery does find the
> sdimage-raspberrypi.wks file to generate the image, as expected, but it
> doesn't show up in 'wic list source-plugins', as you'd expect.

That's expected, there is no separate source plugin for
sdimage-raspberrypi.wks. The kickstart file makes use of
bootimage-partition, so it's effectively almost identical to
sdimage-bootpart.wks but the boot partition size and start offset.

So far the only development boards that will not directly work with
wic's current plugins are Freescale ones (ex. popular iMX6 boards like
Wandboard, Nitrogen6x, Sable Lite etc.). These boards expect u-boot to
be located just after MBR, hence the use case is somewhat similar to
what pcibios source plugin covers.


>
> Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>
>
> > Maciej Borzecki (3):
> >   wic: IMAGE_BOOT_FILES format checks in bootimg-partition source
> >   wic: add globbing support in IMAGE_BOOT_FILES entries
> >   ref-manual: update IMAGE_BOOT_FILES entry
> >
> >  documentation/ref-manual/ref-variables.xml         | 37 +++++++++++++++----
> >  .../lib/wic/plugins/source/bootimg-partition.py    | 43 +++++++++++++++++-----
> >  2 files changed, 64 insertions(+), 16 deletions(-)
> >
> > --
> > 1.9.3
> >
>
>

--
Maciej Borzęcki
Senior Software Developer at Open-RnD Sp. z o.o., Poland
www.open-rnd.pl
mobile: +48 889 117 365, fax: +48 42 657 9079


Niniejsza wiadomość wraz z załącznikami może
zawierać chronione prawem lub poufne informacje i została
wysłana wyłącznie do wiadomości i użytku osób, do których
została zaadresowana. Jeśli wiadomość została otrzymana
przypadkowo zabrania się jej kopiowania lub rozsyłania do osób
trzecich. W takim przypadku uprasza się o natychmiastowe
zniszczenie wiadomości oraz poinformowanie nadawcy o
zaistniałej sytuacji za pomocą wiadomości zwrotnej.
Dziękujemy.

This message, including any attachments hereto,
may contain privileged or confidential information and is sent
solely for the attention and use of the intended addressee(s).
If you are not an intended addressee, you may neither use this
message nor copy or deliver it to anyone. In such case, you
should immediately destroy this message and kindly notify the
sender by reply email. Thank you.

Re: [PATCH 0/3] wic: glob support & minor fixes

[Tom Zanussi]

On Mon, 2014-12-15 at 20:37 +0100, Maciej Borzecki wrote:
> On 12/15 11:04, Tom Zanussi wrote:
> > On Wed, 2014-12-10 at 12:45 +0100, Maciej Borzecki wrote:
> > > This patch series adds changes that enable globbing in IMAGE_BOOT_FILES.
> > > Effectivey this makes use of with with Raspberry Pi a tad easier, when a
> > > lengthy list of boot files can be replaced with just one entry. A related
> > > change to meta-raspberrypi will be sent to the Yocto mailing list.
> > >
> >
> > Looks ok to me, and I tried it out with your raspberrypi layer changes
> > which generated the image (I didn't try any other image types to check
> > against regressions there, though nothing in here looks like it would
> > touch anything related to the other source plugins, so I'm not worried -
> > in the next milestone I hope to have some automated wic testing going to
> > make checking easy).
> >
> > I do think, though, that we need to have the documentation for this in
> > the tool itself.  As it stands, the IMAGE_BOOT_FILES variable is
> > documented in the manual, but even then, there's no context that ties it
> > to the bootimg-partition plugin.  I think we need to add a mechanism so
> > that help text can be added to each plugin, which would be automatically
> > picked up and merged probably into a plugin-specific 'help plugins'
> > section in the tool.
> >
> > The other thing I noticed, which has nothing to do with this patchset,
> > but along the same lines, is that the plugin machinery does find the
> > sdimage-raspberrypi.wks file to generate the image, as expected, but it
> > doesn't show up in 'wic list source-plugins', as you'd expect.
> 
> That's expected, there is no separate source plugin for
> sdimage-raspberrypi.wks. The kickstart file makes use of
> bootimage-partition, so it's effectively almost identical to
> sdimage-bootpart.wks but the boot partition size and start offset.
> 

You're right, I was confusing it with 'list images', which does find the
sdimage-raspberrypi.wks file as expected.

Tom

> So far the only development boards that will not directly work with
> wic's current plugins are Freescale ones (ex. popular iMX6 boards like
> Wandboard, Nitrogen6x, Sable Lite etc.). These boards expect u-boot to
> be located just after MBR, hence the use case is somewhat similar to
> what pcibios source plugin covers.
> 
> 
> >
> > Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>
> >
> > > Maciej Borzecki (3):
> > >   wic: IMAGE_BOOT_FILES format checks in bootimg-partition source
> > >   wic: add globbing support in IMAGE_BOOT_FILES entries
> > >   ref-manual: update IMAGE_BOOT_FILES entry
> > >
> > >  documentation/ref-manual/ref-variables.xml         | 37 +++++++++++++++----
> > >  .../lib/wic/plugins/source/bootimg-partition.py    | 43 +++++++++++++++++-----
> > >  2 files changed, 64 insertions(+), 16 deletions(-)
> > >
> > > --
> > > 1.9.3
> > >
> >
> >
> 
> --
> Maciej Borzęcki
> Senior Software Developer at Open-RnD Sp. z o.o., Poland
> www.open-rnd.pl
> mobile: +48 889 117 365, fax: +48 42 657 9079
> 
> 
> Niniejsza wiadomość wraz z załącznikami może
> zawierać chronione prawem lub poufne informacje i została
> wysłana wyłącznie do wiadomości i użytku osób, do których
> została zaadresowana. Jeśli wiadomość została otrzymana
> przypadkowo zabrania się jej kopiowania lub rozsyłania do osób
> trzecich. W takim przypadku uprasza się o natychmiastowe
> zniszczenie wiadomości oraz poinformowanie nadawcy o
> zaistniałej sytuacji za pomocą wiadomości zwrotnej.
> Dziękujemy.
> 
> This message, including any attachments hereto,
> may contain privileged or confidential information and is sent
> solely for the attention and use of the intended addressee(s).
> If you are not an intended addressee, you may neither use this
> message nor copy or deliver it to anyone. In such case, you
> should immediately destroy this message and kindly notify the
> sender by reply email. Thank you.