[PATCH 0/6] wic: finalize 0.2.0

[PATCH 0/6] wic: finalize 0.2.0

[Ed Bartosh]

Hi,

This patchset includes small bugfixes, code cleanup and usability fixes
for wic release 0.2.0.

Please review.

The following changes since commit a8b723498c9a7106210db140452886894494b4d6:

  bitbake: cooker: Resolve file monitoring race issues when using memres bitbake (2015-08-03 07:36:25 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/0.2.0
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/0.2.0

Ed Bartosh (6):
  wic: improved usage text
  wic: Improve 'wic help overview' content
  wic: Report recipe name for native commands
  wic: code cleanup: remove unused code
  wic: code cleanup: pylint misc.py
  wic: release 0.2.0

 scripts/lib/image/help.py        | 45 ++++++++++++++--------
 scripts/lib/wic/utils/oe/misc.py | 82 +++++++++++++++++++++++-----------------
 scripts/wic                      |  2 +-
 3 files changed, 78 insertions(+), 51 deletions(-)

--
Regards,
Ed

[PATCH 1/6] wic: improved usage text

[Ed Bartosh]

Made command line specification less confusing.
Reformatted usage output.

[YOCTO #7938]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/image/help.py | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index 0666d80..a14ef20 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -101,19 +101,17 @@ wic_usage = """
 
  Create a customized OpenEmbedded image
 
- usage: wic [--version] [--help] COMMAND [ARGS]
+ usage: wic [--version] | [--help] | [COMMAND [ARGS]]
 
  Current 'wic' commands are:
+    help              Show help for command or one of the topics (see below)
     create            Create a new OpenEmbedded image
     list              List available values for options and image properties
 
  Help topics:
-    overview           wic overview - General overview of wic
-    plugins            wic plugins - Overview and API
-    kickstart          wic kickstart - wic kickstart reference
-
- See 'wic help <COMMAND or HELP TOPIC>' for more information on a specific
- command or help topic.
+    overview          wic overview - General overview of wic
+    plugins           wic plugins - Overview and API
+    kickstart         wic kickstart - wic kickstart reference
 """
 
 wic_help_usage = """
-- 
2.1.4

[PATCH 2/6] wic: Improve 'wic help overview' content

[Ed Bartosh]

Included full console output and example of the .wks file
into the 'wic help overview' content.

Used qemux86-64 machine instead of crownbay to make example
working without cloning additional layers.

[YOCTO #7940]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/image/help.py | 33 ++++++++++++++++++++++++---------
 1 file changed, 24 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index a14ef20..5fa5836 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -616,18 +616,33 @@ DESCRIPTION
     and also uses the -o option to have wic create the output
     somewhere other than the default /var/tmp/wic:
 
-      $ wic create ~/test.wks -o /home/trz/testwic --rootfs-dir
-      /home/trz/yocto/build/tmp/work/crownbay/core-image-minimal/1.0-r0/rootfs
-      --bootimg-dir /home/trz/yocto/build/tmp/sysroots/crownbay/usr/share
-      --kernel-dir /home/trz/yocto/build/tmp/sysroots/crownbay/usr/src/kernel
-      --native-sysroot /home/trz/yocto/build/tmp/sysroots/x86_64-linux
+      $ wic create ./test.wks -o ./out --rootfs-dir
+      tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs
+      --bootimg-dir tmp/sysroots/qemux86-64/usr/share
+      --kernel-dir tmp/deploy/images/qemux86-64
+      --native-sysroot tmp/sysroots/x86_64-linux
 
-      Creating image(s)...
+     Creating image(s)...
 
-      Info: The new image(s) can be found here:
-        /home/trz/testwic/build/test-201309260032-sda.direct
+     Info: The new image(s) can be found here:
+           out/build/test-201507211313-sda.direct
+
+     The following build artifacts were used to create the image(s):
+       ROOTFS_DIR:                   tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs
+       BOOTIMG_DIR:                  tmp/sysroots/qemux86-64/usr/share
+       KERNEL_DIR:                   tmp/deploy/images/qemux86-64
+       NATIVE_SYSROOT:               tmp/sysroots/x86_64-linux
+
+     The image(s) were created using OE kickstart file:
+     ./test.wks
+
+     Here is a content of test.wks:
+
+     part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
+     part / --source rootfs --ondisk sda --fstype=ext3 --label platform --align 1024
+
+     bootloader  --timeout=0  --append="rootwait rootfstype=ext3 video=vesafb vga=0x318 console=tty0"
 
-      ...
 
     Finally, here's an example of the actual partition language
     commands used to generate the mkefidisk image i.e. these are the
-- 
2.1.4

[PATCH 3/6] wic: Report recipe name for native commands

[Ed Bartosh]

exec_native_cmd was modified to report recipe to build
native programs.

Pairs executable->recipe are hardcoded as it's not possible
to obtain this information automatically.

[YOCTO #7631]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 29 ++++++++++++++++++++++++++---
 1 file changed, 26 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 4978745..af831d3 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -30,6 +30,21 @@ from collections import defaultdict
 from wic import msger
 from wic.utils import runner
 
+# executable -> recipe pairs for exec_native_cmd
+NATIVE_RECIPES = {"mcopy": "mtools",
+                  "mkdosfs": "dosfstools",
+                  "mkfs.btrfs": "btrfs-tools",
+                  "mkfs.ext2": "e2fsprogs",
+                  "mkfs.ext3": "e2fsprogs",
+                  "mkfs.ext4": "e2fsprogs",
+                  "mkfs.vfat": "dosfstools",
+                  "mksquashfs": "squashfs-tools",
+                  "mkswqp": "util-linux",
+                  "parted": "parted",
+                  "sgdisk": "gptfdisk",
+                  "syslinux": "syslinux"
+                 }
+
 def __exec_cmd(cmd_and_args, as_shell=False, catch=3):
     """
     Execute command, catching stderr, stdout
@@ -85,9 +100,17 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
     rc, out = __exec_cmd(native_cmd_and_args, True, catch)
 
     if rc == 127: # shell command-not-found
-        msger.error("A native program %s required to build the image "
-                    "was not found (see details above). Please make sure "
-                    "it's installed and try again." % args[0])
+        prog = args[0]
+        msg = "A native program %s required to build the image "\
+              "was not found (see details above).\n\n" % prog
+        recipe = NATIVE_RECIPES.get(prog)
+        if recipe:
+            msg += "Please bake it with 'bitbake %s-native' "\
+                   "and try again.\n" % recipe
+        else:
+            msg += "Wic failed to find a recipe to build native %s. Please "\
+                   "file a bug against wic.\n" % prog
+        msger.error(msg)
     if out:
         msger.debug('"%s" output: %s' % (args[0], out))
 
-- 
2.1.4

[PATCH 4/6] wic: code cleanup: remove unused code

[Ed Bartosh]

Removed unused global variable wks_vars and 2 unused functions
get_wks_var and add_wks_var.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 11 -----------
 1 file changed, 11 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index af831d3..0fa8a53 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -120,17 +120,6 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
 
     return (rc, out)
 
-# kickstart doesn't support variable substution in commands, so this
-# is our current simplistic scheme for supporting that
-
-wks_vars = dict()
-
-def get_wks_var(key):
-    return wks_vars[key]
-
-def add_wks_var(key, val):
-    wks_vars[key] = val
-
 BOOTDD_EXTRA_SPACE = 16384
 
 _BITBAKE_VARS = defaultdict(dict)
-- 
2.1.4

[PATCH 5/6] wic: code cleanup: pylint misc.py

[Ed Bartosh]

Fixed pylint warnings.
Increased pylint score from 8.02 to 9.40.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 0fa8a53..d399f2a 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -24,6 +24,7 @@
 # AUTHORS
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
 #
+"""Miscellaneous functions."""
 
 from collections import defaultdict
 
@@ -56,14 +57,14 @@ def __exec_cmd(cmd_and_args, as_shell=False, catch=3):
     msger.debug(args)
 
     if as_shell:
-        rc, out = runner.runtool(cmd_and_args, catch)
+        ret, out = runner.runtool(cmd_and_args, catch)
     else:
-        rc, out = runner.runtool(args, catch)
+        ret, out = runner.runtool(args, catch)
     out = out.strip()
     msger.debug("__exec_cmd: output for %s (rc = %d): %s" % \
-                (cmd_and_args, rc, out))
+                (cmd_and_args, ret, out))
 
-    return (rc, out)
+    return (ret, out)
 
 
 def exec_cmd(cmd_and_args, as_shell=False, catch=3):
@@ -72,10 +73,11 @@ def exec_cmd(cmd_and_args, as_shell=False, catch=3):
 
     Exits if rc non-zero
     """
-    rc, out = __exec_cmd(cmd_and_args, as_shell, catch)
+    ret, out = __exec_cmd(cmd_and_args, as_shell, catch)
 
-    if rc != 0:
-        msger.error("exec_cmd: %s returned '%s' instead of 0" % (cmd_and_args, rc))
+    if ret != 0:
+        msger.error("exec_cmd: %s returned '%s' instead of 0" % \
+                    (cmd_and_args, ret))
 
     return out
 
@@ -97,9 +99,9 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
     args = cmd_and_args.split()
     msger.debug(args)
 
-    rc, out = __exec_cmd(native_cmd_and_args, True, catch)
+    ret, out = __exec_cmd(native_cmd_and_args, True, catch)
 
-    if rc == 127: # shell command-not-found
+    if ret == 127: # shell command-not-found
         prog = args[0]
         msg = "A native program %s required to build the image "\
               "was not found (see details above).\n\n" % prog
@@ -114,11 +116,11 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
     if out:
         msger.debug('"%s" output: %s' % (args[0], out))
 
-    if rc != 0:
+    if ret != 0:
         msger.error("exec_cmd: '%s' returned '%s' instead of 0" % \
-                    (cmd_and_args, rc))
+                    (cmd_and_args, ret))
 
-    return (rc, out)
+    return ret, out
 
 BOOTDD_EXTRA_SPACE = 16384
 
@@ -137,10 +139,10 @@ def get_bitbake_var(var, image=None):
 
         log_level = msger.get_loglevel()
         msger.set_loglevel('normal')
-        rc, lines = __exec_cmd(cmd)
+        ret, lines = __exec_cmd(cmd)
         msger.set_loglevel(log_level)
 
-        if rc:
+        if ret:
             print "Couldn't get '%s' output." % cmd
             print "Bitbake failed with error:\n%s\n" % lines
             return
@@ -176,14 +178,14 @@ def parse_sourceparams(sourceparams):
 
     params = sourceparams.split(',')
     if params:
-        for p in params:
-            if not p:
+        for par in params:
+            if not par:
                 continue
-            if not '=' in p:
-                key = p
+            if not '=' in par:
+                key = par
                 val = None
             else:
-                key, val = p.split('=')
+                key, val = par.split('=')
             params_dict[key] = val
 
     return params_dict
-- 
2.1.4

[PATCH 6/6] wic: release 0.2.0

[Ed Bartosh]

Major changes in 0.2.0:
- implemented UUID support
- fixed support of GPT partition tables
- implemented running bitbake from wic
- implemented image compressing
- started testing with oe-selftest. implemented 17 test cases
- used native tools (parted, mkfs, mtools, etc)
- usability and documentation fixes
- code clenup

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/wic | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/wic b/scripts/wic
index 9435ce0..c1d3003 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -29,7 +29,7 @@
 # Tom Zanussi <tom.zanussi (at] linux.intel.com>
 #
 
-__version__ = "0.1.0"
+__version__ = "0.2.0"
 
 # Python Standard Library modules
 import os
-- 
2.1.4