[PATCH 1/6][dizzy] wic: Don't allow mkfs to fail silently in partition command

[PATCH 1/6][dizzy] wic: Don't allow mkfs to fail silently in partition command

[Tom Zanussi]

The return code from the mkfs command used by the partition creation
command was being ignored, allowing it to silently fail and leaving
users mystified as to why the resulting filesystem was corrupted.

This became obvious when failures occurred when creating large
e.g. sdk filesystems [YOCTO #6863].

(From OE-Core rev: 8cef3b06f7e9f9d922673f430ddb3170d2fac000)

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../lib/wic/kickstart/custom_commands/partition.py | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/wic/kickstart/custom_commands/partition.py b/scripts/lib/wic/kickstart/custom_commands/partition.py
index 8b0ace6..3950b43 100644
--- a/scripts/lib/wic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/wic/kickstart/custom_commands/partition.py
@@ -241,8 +241,10 @@ class Wic_PartData(Mic_PartData):
 
         mkfs_cmd = "mkfs.%s -F %s %s -d %s" % \
             (self.fstype, extra_imagecmd, rootfs, image_rootfs)
-        exec_native_cmd(pseudo + mkfs_cmd, native_sysroot)
-
+        (rc, out) = exec_native_cmd(pseudo + mkfs_cmd, native_sysroot)
+        if rc:
+            print "rootfs_dir: %s" % rootfs_dir
+            msger.error("ERROR: mkfs.%s returned '%s' instead of 0 (which you probably don't want to ignore, use --debug for details) when creating filesystem from rootfs directory: %s" % (self.fstype, rc, rootfs_dir))
 
         # get the rootfs size in the right units for kickstart (Mb)
         du_cmd = "du -Lbms %s" % rootfs
@@ -284,7 +286,9 @@ class Wic_PartData(Mic_PartData):
 
         mkfs_cmd = "mkfs.%s -b %d -r %s %s" % \
             (self.fstype, rootfs_size * 1024, image_rootfs, rootfs)
-        exec_native_cmd(pseudo + mkfs_cmd, native_sysroot)
+        (rc, out) = exec_native_cmd(pseudo + mkfs_cmd, native_sysroot)
+        if rc:
+            msger.error("ERROR: mkfs.%s returned '%s' instead of 0 (which you probably don't want to ignore, use --debug for details) when creating filesystem from rootfs directory: %s" % (self.fstype, rc, rootfs_dir))
 
         # get the rootfs size in the right units for kickstart (Mb)
         du_cmd = "du -Lbms %s" % rootfs
@@ -396,7 +400,9 @@ class Wic_PartData(Mic_PartData):
         extra_imagecmd = "-i 8192"
 
         mkfs_cmd = "mkfs.%s -F %s %s" % (self.fstype, extra_imagecmd, fs)
-        exec_native_cmd(mkfs_cmd, native_sysroot)
+        (rc, out) = exec_native_cmd(mkfs_cmd, native_sysroot)
+        if rc:
+            msger.error("ERROR: mkfs.%s returned '%s' instead of 0 (which you probably don't want to ignore, use --debug for details)" % (self.fstype, rc))
 
         self.source_file = fs
 
@@ -414,10 +420,14 @@ class Wic_PartData(Mic_PartData):
         exec_cmd(dd_cmd)
 
         mkfs_cmd = "mkfs.%s -b %d %s" % (self.fstype, self.size * 1024, rootfs)
-        exec_native_cmd(mkfs_cmd, native_sysroot)
+        (rc, out) = exec_native_cmd(mkfs_cmd, native_sysroot)
+        if rc:
+            msger.error("ERROR: mkfs.%s returned '%s' instead of 0 (which you probably don't want to ignore, use --debug for details)" % (self.fstype, rc))
 
         mkfs_cmd = "mkfs.%s -F %s %s" % (self.fstype, extra_imagecmd, fs)
-        exec_native_cmd(mkfs_cmd, native_sysroot)
+        (rc, out) = exec_native_cmd(mkfs_cmd, native_sysroot)
+        if rc:
+            msger.error("ERROR: mkfs.%s returned '%s' instead of 0 (which you probably don't want to ignore, use --debug for details)" % (self.fstype, rc))
 
         self.source_file = fs
 
-- 
1.9.3

[PATCH 2/6][dizzy] wic: Use overhead factor when creating partitions from rootfs directories

[Tom Zanussi]

When creating partitions sized to given rootfs directories, filesystem
creation could fail in cases where the calculated target partition
size was too small to contain the filesystem created using mkfs.  This
occurred in particular when creating partitions to contain very large
filesystems such as those containing sdk image artifacts.

This same limition is present in the oe-core image creation classes,
which can be readily see by changing IMAGE_OVERHEAD_FACTOR from the
default 1.3 to 1.0 and building a sato-sdk image.

It should be possible to calculate required sizes exactly given the
source rootfs and target filesystem types, but for now, to address the
specific problem users are hitting in such situations, we'll just do
exactly what oe-core does and define and use an IMAGE_OVERHEAD_FACTOR
or 1.3 in those cases.

Fixes [YOCTO #6863].

(From OE-Core rev: bbaef3ff5833fc1d97b7b028d7770834f62789da)

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/lib/wic/kickstart/custom_commands/partition.py | 2 ++
 scripts/lib/wic/utils/oe/misc.py                       | 1 +
 2 files changed, 3 insertions(+)

diff --git a/scripts/lib/wic/kickstart/custom_commands/partition.py b/scripts/lib/wic/kickstart/custom_commands/partition.py
index 3950b43..54a494e 100644
--- a/scripts/lib/wic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/wic/kickstart/custom_commands/partition.py
@@ -229,6 +229,7 @@ class Wic_PartData(Mic_PartData):
             extra_blocks = IMAGE_EXTRA_SPACE
 
         rootfs_size = actual_rootfs_size + extra_blocks
+        rootfs_size *= IMAGE_OVERHEAD_FACTOR
 
         msger.debug("Added %d extra blocks to %s to get to %d total blocks" % \
                     (extra_blocks, self.mountpoint, rootfs_size))
@@ -276,6 +277,7 @@ class Wic_PartData(Mic_PartData):
             extra_blocks = IMAGE_EXTRA_SPACE
 
         rootfs_size = actual_rootfs_size + extra_blocks
+        rootfs_size *= IMAGE_OVERHEAD_FACTOR
 
         msger.debug("Added %d extra blocks to %s to get to %d total blocks" % \
                     (extra_blocks, self.mountpoint, rootfs_size))
diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index aa9b235..b0b5baa 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -123,6 +123,7 @@ def add_wks_var(key, val):
 
 BOOTDD_EXTRA_SPACE = 16384
 IMAGE_EXTRA_SPACE = 10240
+IMAGE_OVERHEAD_FACTOR = 1.3
 
 __bitbake_env_lines = ""
 
-- 
1.9.3

[PATCH 0/6][dizzy] some wic bugfixes

[Tom Zanussi]

From the original posting:

This patchset fixes a couple serious wic bugs:

[YOCTO #6863] - Wic fails to create a working image for a large file system
[YOCTO #6290] - Update WIC for gummiboot

I generated core-image-sdk images for each of the wic image types and
was able to fsck/mount the resulting filesystems.

These should be applied to the next dizzy point release.

The following changes since commit e69653b01dd8ad9fda2355b8112a65c813ce7fd5:

  tzdata: update to 2014j (2014-11-17 08:01:21 -0800)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib.git tzanussi/dizzy-next
  http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/dizzy-next

Tom Zanussi (6):
  wic: Don't allow mkfs to fail silently in partition command
  wic: Use overhead factor when creating partitions from rootfs
    directories
  wic: Update the help text to include -D (--debug)
  Revert "wic: set bootimg_dir when using image-name artifacts"
  wic: Remove special-case bootimg_dir
  wic: Update bootimg-partition to use bootimg_dir

 scripts/lib/image/engine.py                        | 42 +++++-----------------
 scripts/lib/image/help.py                          | 23 ++++++++----
 scripts/lib/wic/imager/direct.py                   |  5 +--
 .../lib/wic/kickstart/custom_commands/partition.py | 24 +++++++++----
 scripts/lib/wic/plugins/imager/direct_plugin.py    | 20 +++++------
 scripts/lib/wic/plugins/source/bootimg-efi.py      |  5 ++-
 .../lib/wic/plugins/source/bootimg-partition.py    |  9 +++--
 scripts/lib/wic/plugins/source/bootimg-pcbios.py   |  3 +-
 scripts/lib/wic/utils/oe/misc.py                   |  1 +
 scripts/wic                                        | 13 +++----
 10 files changed, 68 insertions(+), 77 deletions(-)

-- 
1.9.3

[PATCH 3/6][dizzy] wic: Update the help text to include -D (--debug)

[Tom Zanussi]

The --debug option is missing from the wic help text; this adds it and
at the same time rearranges the usage into a more logical arrangement.

(From OE-Core rev: cf5144ef241d8f4ccaa3461ae5c9f89c2cf2f8d1)

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/lib/image/help.py | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index 080795e..0963532 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -109,8 +109,9 @@ wic_create_usage = """
 
  usage: wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>]
             [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
-            [-e | --image-name] [-r, --rootfs-dir] [-b, --bootimg-dir]
-            [-k, --kernel-dir] [-n, --native-sysroot] [-s, --skip-build-check]
+            [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
+            [-r, --rootfs-dir] [-b, --bootimg-dir]
+            [-k, --kernel-dir] [-n, --native-sysroot]
 
  This command creates an OpenEmbedded image based on the 'OE kickstart
  commands' found in the <wks file>.
@@ -129,8 +130,9 @@ NAME
 SYNOPSIS
     wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>]
         [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
-        [-e | --image-name] [-r, --rootfs-dir] [-b, --bootimg-dir]
-        [-k, --kernel-dir] [-n, --native-sysroot] [-s, --skip-build-check]
+        [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
+        [-r, --rootfs-dir] [-b, --bootimg-dir]
+        [-k, --kernel-dir] [-n, --native-sysroot]
 
 DESCRIPTION
     This command creates an OpenEmbedded image based on the 'OE
@@ -172,6 +174,12 @@ DESCRIPTION
     explicitly, 'wic' assumes the user knows what he or she is doing
     and skips the build check.
 
+    The -D option is used to display debug information detailing
+    exactly what happens behind the scenes when a create request is
+    fulfilled (or not, as the case may be).  It enumerates and
+    displays the command sequence used, and should be included in any
+    bug report describing unexpected results.
+
     When 'wic -e' is used, the locations for the build artifacts
     values are determined by 'wic -e' from the output of the 'bitbake
     -e' command given an image name e.g. 'core-image-minimal' and a
@@ -519,8 +527,9 @@ DESCRIPTION
 
        usage: wic create <wks file or image name> [-o <DIRNAME> | ...]
             [-i <JSON PROPERTY FILE> | --infile <JSON PROPERTY_FILE>]
-            [-e | --image-name] [-r, --rootfs-dir] [-b, --bootimg-dir]
-            [-k, --kernel-dir] [-n, --native-sysroot] [-s, --skip-build-check]
+            [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
+            [-r, --rootfs-dir] [-b, --bootimg-dir]
+            [-k, --kernel-dir] [-n, --native-sysroot]
 
        This command creates an OpenEmbedded image based on the 'OE
        kickstart commands' found in the <wks file>.
-- 
1.9.3

[PATCH 4/6][dizzy] Revert "wic: set bootimg_dir when using image-name artifacts"

[Tom Zanussi]

This reverts commit 7ce1dc13f91df70e8a2f420e7c3eba51cbc4bd48.

This patch broke the assumption that a non-null boot_dir means a
user-assigned (-b command-line param) value.

Reverting doesn't break anything, since the case it was added for
doesn't use the boot_dir for anything except debugging anyhow.

Fixes [YOCTO #6290]

(From OE-Core rev: db90f10bf31dec8d7d7bb2d3680d50e133662850)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/lib/image/engine.py | 9 ++-------
 scripts/wic                 | 7 +++----
 2 files changed, 5 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/image/engine.py b/scripts/lib/image/engine.py
index 3813fec..f1df8b4 100644
--- a/scripts/lib/image/engine.py
+++ b/scripts/lib/image/engine.py
@@ -67,8 +67,7 @@ def find_artifacts(image_name):
     """
     bitbake_env_lines = get_bitbake_env_lines()
 
-    rootfs_dir = kernel_dir = bootimg_dir = ""
-    hdddir = staging_data_dir = native_sysroot = ""
+    rootfs_dir = kernel_dir = hdddir = staging_data_dir = native_sysroot = ""
 
     for line in bitbake_env_lines.split('\n'):
         if (get_line_val(line, "IMAGE_ROOTFS")):
@@ -86,12 +85,8 @@ def find_artifacts(image_name):
         if (get_line_val(line, "STAGING_DIR_NATIVE")):
             native_sysroot = get_line_val(line, "STAGING_DIR_NATIVE")
             continue
-        if (get_line_val(line, "DEPLOY_DIR_IMAGE")):
-            bootimg_dir = get_line_val(line, "DEPLOY_DIR_IMAGE")
-            continue
 
-    return (rootfs_dir, kernel_dir, bootimg_dir, hdddir, staging_data_dir, \
-            native_sysroot)
+    return (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot)
 
 
 CANNED_IMAGE_DIR = "lib/image/canned-wks" # relative to scripts
diff --git a/scripts/wic b/scripts/wic
index 7314810..15cc9b3 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -134,8 +134,8 @@ def wic_create_subcommand(args, usage_str):
     bootimg_dir = staging_data_dir = hdddir = ""
 
     if options.image_name:
-        (rootfs_dir, kernel_dir, bootimg_dir, hdddir, \
-         staging_data_dir, native_sysroot) = find_artifacts(options.image_name)
+        (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot) = \
+            find_artifacts(options.image_name)
 
     wks_file = args[0]
 
@@ -172,8 +172,7 @@ def wic_create_subcommand(args, usage_str):
         not_found = not_found_dir = ""
         if not os.path.isdir(rootfs_dir):
             (not_found, not_found_dir) = ("rootfs-dir", rootfs_dir)
-        elif not os.path.isdir(bootimg_dir) and not os.path.isdir(hdddir) \
-             and not os.path.isdir(staging_data_dir):
+        elif not os.path.isdir(hdddir) and not os.path.isdir(staging_data_dir):
             (not_found, not_found_dir) = ("bootimg-dir", bootimg_dir)
         elif not os.path.isdir(kernel_dir):
             (not_found, not_found_dir) = ("kernel-dir", kernel_dir)
-- 
1.9.3

[PATCH 5/6][dizzy] wic: Remove special-case bootimg_dir

[Tom Zanussi]

The first iterations of wic very shortsightedly catered to two
specific use-cases and added special-purpose params for those cases so
that they could be directly given their corresponding boot artifacts.
(hdddir and staging_data_dir).

As more use-cases are added, it becomes rather obvious that such a
scheme doens't scale, and additionally causes confusion for plugin
writers.

This removes those special cases and states explicitly in the help
text that plugins are responsible for locating their own boot
artifacts.

(From OE-Core rev: 6ba3eb5ff7c47aee6b3419fb3a348a634fe74ac9)

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/lib/image/engine.py                      | 37 ++++++------------------
 scripts/lib/image/help.py                        |  2 +-
 scripts/lib/wic/imager/direct.py                 |  5 +---
 scripts/lib/wic/plugins/imager/direct_plugin.py  | 20 +++++--------
 scripts/lib/wic/plugins/source/bootimg-efi.py    |  5 ++--
 scripts/lib/wic/plugins/source/bootimg-pcbios.py |  3 +-
 scripts/wic                                      | 12 ++++----
 7 files changed, 27 insertions(+), 57 deletions(-)

diff --git a/scripts/lib/image/engine.py b/scripts/lib/image/engine.py
index f1df8b4..e794545 100644
--- a/scripts/lib/image/engine.py
+++ b/scripts/lib/image/engine.py
@@ -67,7 +67,7 @@ def find_artifacts(image_name):
     """
     bitbake_env_lines = get_bitbake_env_lines()
 
-    rootfs_dir = kernel_dir = hdddir = staging_data_dir = native_sysroot = ""
+    rootfs_dir = kernel_dir = bootimg_dir = native_sysroot = ""
 
     for line in bitbake_env_lines.split('\n'):
         if (get_line_val(line, "IMAGE_ROOTFS")):
@@ -76,17 +76,11 @@ def find_artifacts(image_name):
         if (get_line_val(line, "STAGING_KERNEL_DIR")):
             kernel_dir = get_line_val(line, "STAGING_KERNEL_DIR")
             continue
-        if (get_line_val(line, "HDDDIR")):
-            hdddir = get_line_val(line, "HDDDIR")
-            continue
-        if (get_line_val(line, "STAGING_DATADIR")):
-            staging_data_dir = get_line_val(line, "STAGING_DATADIR")
-            continue
         if (get_line_val(line, "STAGING_DIR_NATIVE")):
             native_sysroot = get_line_val(line, "STAGING_DIR_NATIVE")
             continue
 
-    return (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot)
+    return (rootfs_dir, kernel_dir, bootimg_dir, native_sysroot)
 
 
 CANNED_IMAGE_DIR = "lib/image/canned-wks" # relative to scripts
@@ -185,18 +179,15 @@ def list_source_plugins():
 
 
 def wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
-               native_sysroot, hdddir, staging_data_dir, scripts_path,
-               image_output_dir, debug, properties_file, properties=None):
-    """
-    Create image
+               native_sysroot, scripts_path, image_output_dir, debug,
+               properties_file, properties=None):
+    """Create image
 
     wks_file - user-defined OE kickstart file
     rootfs_dir - absolute path to the build's /rootfs dir
     bootimg_dir - absolute path to the build's boot artifacts directory
     kernel_dir - absolute path to the build's kernel directory
     native_sysroot - absolute path to the build's native sysroots dir
-    hdddir - absolute path to the build's HDDDIR dir
-    staging_data_dir - absolute path to the build's STAGING_DATA_DIR dir
     scripts_path - absolute path to /scripts dir
     image_output_dir - dirname to create for image
     properties_file - use values from this file if nonempty i.e no prompting
@@ -211,22 +202,14 @@ def wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
     rootfs_dir:        IMAGE_ROOTFS
     kernel_dir:        STAGING_KERNEL_DIR
     native_sysroot:    STAGING_DIR_NATIVE
-    hdddir:            HDDDIR
-    staging_data_dir:  STAGING_DATA_DIR
 
-    In the above case, bootimg_dir remains unset and the image
-    creation code determines which of the passed-in directories to
-    use.
+    In the above case, bootimg_dir remains unset and the
+    plugin-specific image creation code is responsible for finding the
+    bootimg artifacts.
 
     In the case where the values are passed in explicitly i.e 'wic -e'
     is not used but rather the individual 'wic' options are used to
-    explicitly specify these values, hdddir and staging_data_dir will
-    be unset, but bootimg_dir must be explicit i.e. explicitly set to
-    either hdddir or staging_data_dir, depending on the image being
-    generated.  The other values (rootfs_dir, kernel_dir, and
-    native_sysroot) correspond to the same values found above via
-    'bitbake -e').
-
+    explicitly specify these values.
     """
     try:
         oe_builddir = os.environ["BUILDDIR"]
@@ -242,8 +225,6 @@ def wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
     direct_args.insert(0, bootimg_dir)
     direct_args.insert(0, kernel_dir)
     direct_args.insert(0, native_sysroot)
-    direct_args.insert(0, hdddir)
-    direct_args.insert(0, staging_data_dir)
     direct_args.insert(0, "direct")
 
     if debug:
diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index 0963532..6b74f57 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -189,7 +189,7 @@ DESCRIPTION
     -r:        IMAGE_ROOTFS
     -k:        STAGING_KERNEL_DIR
     -n:        STAGING_DIR_NATIVE
-    -b:        HDDDIR and STAGING_DATA_DIR (handlers decide which to use)
+    -b:        empty (plugin-specific handlers must determine this)
 
     If 'wic -e' is not used, the user needs to select the appropriate
     value for -b (as well as -r, -k, and -n).
diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 5b12856..6b2ab33 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -52,8 +52,7 @@ class DirectImageCreator(BaseImageCreator):
     """
 
     def __init__(self, oe_builddir, image_output_dir, rootfs_dir, bootimg_dir,
-                 kernel_dir, native_sysroot, hdddir, staging_data_dir,
-                 creatoropts=None):
+                 kernel_dir, native_sysroot, creatoropts=None):
         """
         Initialize a DirectImageCreator instance.
 
@@ -74,8 +73,6 @@ class DirectImageCreator(BaseImageCreator):
         self.bootimg_dir = bootimg_dir
         self.kernel_dir = kernel_dir
         self.native_sysroot = native_sysroot
-        self.hdddir = hdddir
-        self.staging_data_dir = staging_data_dir
 
     def __write_fstab(self, image_rootfs):
         """overriden to generate fstab (temporarily) in rootfs. This is called
diff --git a/scripts/lib/wic/plugins/imager/direct_plugin.py b/scripts/lib/wic/plugins/imager/direct_plugin.py
index dabd6fc..5601c3f 100644
--- a/scripts/lib/wic/plugins/imager/direct_plugin.py
+++ b/scripts/lib/wic/plugins/imager/direct_plugin.py
@@ -58,21 +58,19 @@ class DirectPlugin(ImagerPlugin):
         """
         Create direct image, called from creator as 'direct' cmd
         """
-        if len(args) != 9:
+        if len(args) != 7:
             raise errors.Usage("Extra arguments given")
 
-        staging_data_dir = args[0]
-        hdddir = args[1]
-        native_sysroot = args[2]
-        kernel_dir = args[3]
-        bootimg_dir = args[4]
-        rootfs_dir = args[5]
+        native_sysroot = args[0]
+        kernel_dir = args[1]
+        bootimg_dir = args[2]
+        rootfs_dir = args[3]
 
         creatoropts = configmgr.create
-        ksconf = args[6]
+        ksconf = args[4]
 
-        image_output_dir = args[7]
-        oe_builddir = args[8]
+        image_output_dir = args[5]
+        oe_builddir = args[6]
 
         krootfs_dir = self.__rootfs_dir_to_dict(rootfs_dir)
 
@@ -84,8 +82,6 @@ class DirectPlugin(ImagerPlugin):
                                             bootimg_dir,
                                             kernel_dir,
                                             native_sysroot,
-                                            hdddir,
-                                            staging_data_dir,
                                             creatoropts)
 
         try:
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 855bbc2..e4067b6 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -173,7 +173,6 @@ class BootimgEFIPlugin(SourcePlugin):
             cr.set_bootimg_dir(bootimg_dir)
 
         staging_kernel_dir = kernel_dir
-        staging_data_dir = bootimg_dir
 
         hdddir = "%s/hdd/boot" % cr_workdir
 
@@ -185,12 +184,12 @@ class BootimgEFIPlugin(SourcePlugin):
             if source_params['loader'] == 'grub-efi':
                 shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir,
                                 "%s/grub.cfg" % cr_workdir)
-                cp_cmd = "cp %s/EFI/BOOT/* %s/EFI/BOOT" % (staging_data_dir, hdddir)
+                cp_cmd = "cp %s/EFI/BOOT/* %s/EFI/BOOT" % (bootimg_dir, hdddir)
                 exec_cmd(cp_cmd, True)
                 shutil.move("%s/grub.cfg" % cr_workdir,
                             "%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir)
             elif source_params['loader'] == 'gummiboot':
-                cp_cmd = "cp %s/EFI/BOOT/* %s/EFI/BOOT" % (staging_data_dir, hdddir)
+                cp_cmd = "cp %s/EFI/BOOT/* %s/EFI/BOOT" % (bootimg_dir, hdddir)
                 exec_cmd(cp_cmd, True)
             else:
                 msger.error("unrecognized bootimg-efi loader: %s" % source_params['loader'])
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index 6057bab..8a1aca1 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -144,7 +144,6 @@ class BootimgPcbiosPlugin(SourcePlugin):
             cr.set_bootimg_dir(bootimg_dir)
 
         staging_kernel_dir = kernel_dir
-        staging_data_dir = bootimg_dir
 
         hdddir = "%s/hdd/boot" % cr_workdir
 
@@ -153,7 +152,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
         exec_cmd(install_cmd)
 
         install_cmd = "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" \
-            % (staging_data_dir, hdddir)
+            % (bootimg_dir, hdddir)
         exec_cmd(install_cmd)
 
         du_cmd = "du -bks %s" % hdddir
diff --git a/scripts/wic b/scripts/wic
index 15cc9b3..e7df60f 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -131,11 +131,11 @@ def wic_create_subcommand(args, usage_str):
         sys.exit(1)
     set_bitbake_env_lines(bitbake_env_lines)
 
-    bootimg_dir = staging_data_dir = hdddir = ""
+    bootimg_dir = ""
 
     if options.image_name:
-        (rootfs_dir, kernel_dir, hdddir, staging_data_dir, native_sysroot) = \
-            find_artifacts(options.image_name)
+        (rootfs_dir, kernel_dir, bootimg_dir, native_sysroot) \
+            = find_artifacts(options.image_name)
 
     wks_file = args[0]
 
@@ -172,8 +172,6 @@ def wic_create_subcommand(args, usage_str):
         not_found = not_found_dir = ""
         if not os.path.isdir(rootfs_dir):
             (not_found, not_found_dir) = ("rootfs-dir", rootfs_dir)
-        elif not os.path.isdir(hdddir) and not os.path.isdir(staging_data_dir):
-            (not_found, not_found_dir) = ("bootimg-dir", bootimg_dir)
         elif not os.path.isdir(kernel_dir):
             (not_found, not_found_dir) = ("kernel-dir", kernel_dir)
         elif not os.path.isdir(native_sysroot):
@@ -197,8 +195,8 @@ def wic_create_subcommand(args, usage_str):
     rootfs_dir = rootfs_dir_to_args(krootfs_dir)
 
     wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
-               native_sysroot, hdddir, staging_data_dir, scripts_path,
-               image_output_dir, options.debug, options.properties_file)
+               native_sysroot, scripts_path, image_output_dir,
+               options.debug, options.properties_file)
 
 
 def wic_list_subcommand(args, usage_str):
-- 
1.9.3

[PATCH 6/6][dizzy] wic: Update bootimg-partition to use bootimg_dir

[Tom Zanussi]

Update bootimg-partition to use bootimg_dir instead of img_deploy_dir,
to match similar usage in other plugins.

As mentioned elsewhere, plugins should use the passed-in value for
bootimg_dir directly if non-null, which corresponds to a user-assigned
value specified via a -b command-line param, and only fetch the value
from bitbake if that value is null.

(From OE-Core rev: 3822f8a7b33da56ecd9144b4bcae50734fb1af81)

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 scripts/lib/wic/plugins/source/bootimg-partition.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-partition.py b/scripts/lib/wic/plugins/source/bootimg-partition.py
index cc72b2f..abf2494 100644
--- a/scripts/lib/wic/plugins/source/bootimg-partition.py
+++ b/scripts/lib/wic/plugins/source/bootimg-partition.py
@@ -71,8 +71,13 @@ class BootimgPartitionPlugin(SourcePlugin):
         install_cmd = "install -d %s" % hdddir
         exec_cmd(install_cmd)
 
+        if not bootimg_dir:
+            bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+            if not bootimg_dir:
+                msger.error("Couldn't find DEPLOY_DIR_IMAGE, exiting\n")
+
         msger.debug('Bootimg dir: %s' % bootimg_dir)
-        img_deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+
         boot_files = get_bitbake_var("IMAGE_BOOT_FILES")
 
         if not boot_files:
@@ -93,7 +98,7 @@ class BootimgPartitionPlugin(SourcePlugin):
 
         for deploy_entry in deploy_files:
             src, dst = deploy_entry
-            src_path = os.path.join(img_deploy_dir, src)
+            src_path = os.path.join(bootimg_dir, src)
             dst_path = os.path.join(hdddir, dst)
 
             msger.debug('Install %s as %s' % (os.path.basename(src_path),
-- 
1.9.3

Re: [PATCH 0/6][dizzy] some wic bugfixes

[akuster808]

Thanks Tom,

I will stage these along with a security fix.

- Armin

On 11/21/2014 08:07 AM, Tom Zanussi wrote:
>  From the original posting:
>
> This patchset fixes a couple serious wic bugs:
>
> [YOCTO #6863] - Wic fails to create a working image for a large file system
> [YOCTO #6290] - Update WIC for gummiboot
>
> I generated core-image-sdk images for each of the wic image types and
> was able to fsck/mount the resulting filesystems.
>
> These should be applied to the next dizzy point release.
>
> The following changes since commit e69653b01dd8ad9fda2355b8112a65c813ce7fd5:
>
>    tzdata: update to 2014j (2014-11-17 08:01:21 -0800)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib.git tzanussi/dizzy-next
>    http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/dizzy-next
>
> Tom Zanussi (6):
>    wic: Don't allow mkfs to fail silently in partition command
>    wic: Use overhead factor when creating partitions from rootfs
>      directories
>    wic: Update the help text to include -D (--debug)
>    Revert "wic: set bootimg_dir when using image-name artifacts"
>    wic: Remove special-case bootimg_dir
>    wic: Update bootimg-partition to use bootimg_dir
>
>   scripts/lib/image/engine.py                        | 42 +++++-----------------
>   scripts/lib/image/help.py                          | 23 ++++++++----
>   scripts/lib/wic/imager/direct.py                   |  5 +--
>   .../lib/wic/kickstart/custom_commands/partition.py | 24 +++++++++----
>   scripts/lib/wic/plugins/imager/direct_plugin.py    | 20 +++++------
>   scripts/lib/wic/plugins/source/bootimg-efi.py      |  5 ++-
>   .../lib/wic/plugins/source/bootimg-partition.py    |  9 +++--
>   scripts/lib/wic/plugins/source/bootimg-pcbios.py   |  3 +-
>   scripts/lib/wic/utils/oe/misc.py                   |  1 +
>   scripts/wic                                        | 13 +++----
>   10 files changed, 68 insertions(+), 77 deletions(-)
>

Re: [PATCH 0/6][dizzy] some wic bugfixes

[Philip Balister]

On 11/21/2014 11:07 AM, Tom Zanussi wrote:
>>From the original posting:
> 
> This patchset fixes a couple serious wic bugs:
> 
> [YOCTO #6863] - Wic fails to create a working image for a large file system
> [YOCTO #6290] - Update WIC for gummiboot
> 
> I generated core-image-sdk images for each of the wic image types and
> was able to fsck/mount the resulting filesystems.
> 
> These should be applied to the next dizzy point release.
> 
> The following changes since commit e69653b01dd8ad9fda2355b8112a65c813ce7fd5:
> 
>   tzdata: update to 2014j (2014-11-17 08:01:21 -0800)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib.git tzanussi/dizzy-next
>   http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/dizzy-next
> 
> Tom Zanussi (6):
>   wic: Don't allow mkfs to fail silently in partition command
>   wic: Use overhead factor when creating partitions from rootfs
>     directories

It looks like all the patches in this request made it into dizzy, except
the overhead factor one. Is there a reason for this? Can we get the
overhead one into dizzy also?

Philip




>   wic: Update the help text to include -D (--debug)
>   Revert "wic: set bootimg_dir when using image-name artifacts"
>   wic: Remove special-case bootimg_dir
>   wic: Update bootimg-partition to use bootimg_dir
> 
>  scripts/lib/image/engine.py                        | 42 +++++-----------------
>  scripts/lib/image/help.py                          | 23 ++++++++----
>  scripts/lib/wic/imager/direct.py                   |  5 +--
>  .../lib/wic/kickstart/custom_commands/partition.py | 24 +++++++++----
>  scripts/lib/wic/plugins/imager/direct_plugin.py    | 20 +++++------
>  scripts/lib/wic/plugins/source/bootimg-efi.py      |  5 ++-
>  .../lib/wic/plugins/source/bootimg-partition.py    |  9 +++--
>  scripts/lib/wic/plugins/source/bootimg-pcbios.py   |  3 +-
>  scripts/lib/wic/utils/oe/misc.py                   |  1 +
>  scripts/wic                                        | 13 +++----
>  10 files changed, 68 insertions(+), 77 deletions(-)
> 

Re: [PATCH 0/6][dizzy] some wic bugfixes

[akuster808]

On 12/03/2014 05:40 AM, Philip Balister wrote:
> On 11/21/2014 11:07 AM, Tom Zanussi wrote:
>> >From the original posting:
>>
>> This patchset fixes a couple serious wic bugs:
>>
>> [YOCTO #6863] - Wic fails to create a working image for a large file system
>> [YOCTO #6290] - Update WIC for gummiboot
>>
>> I generated core-image-sdk images for each of the wic image types and
>> was able to fsck/mount the resulting filesystems.
>>
>> These should be applied to the next dizzy point release.
>>
>> The following changes since commit e69653b01dd8ad9fda2355b8112a65c813ce7fd5:
>>
>>    tzdata: update to 2014j (2014-11-17 08:01:21 -0800)
>>
>> are available in the git repository at:
>>
>>    git://git.yoctoproject.org/poky-contrib.git tzanussi/dizzy-next
>>    http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/dizzy-next
>>
>> Tom Zanussi (6):
>>    wic: Don't allow mkfs to fail silently in partition command
>>    wic: Use overhead factor when creating partitions from rootfs
>>      directories
>
> It looks like all the patches in this request made it into dizzy, except
> the overhead factor one.

Then its not technically _all_ then ; )

Is there a reason for this?
No reason. just missed it

Can we get the
> overhead one into dizzy also?

sure.

- Armin

>
> Philip
>
>
>
>
>>    wic: Update the help text to include -D (--debug)
>>    Revert "wic: set bootimg_dir when using image-name artifacts"
>>    wic: Remove special-case bootimg_dir
>>    wic: Update bootimg-partition to use bootimg_dir
>>
>>   scripts/lib/image/engine.py                        | 42 +++++-----------------
>>   scripts/lib/image/help.py                          | 23 ++++++++----
>>   scripts/lib/wic/imager/direct.py                   |  5 +--
>>   .../lib/wic/kickstart/custom_commands/partition.py | 24 +++++++++----
>>   scripts/lib/wic/plugins/imager/direct_plugin.py    | 20 +++++------
>>   scripts/lib/wic/plugins/source/bootimg-efi.py      |  5 ++-
>>   .../lib/wic/plugins/source/bootimg-partition.py    |  9 +++--
>>   scripts/lib/wic/plugins/source/bootimg-pcbios.py   |  3 +-
>>   scripts/lib/wic/utils/oe/misc.py                   |  1 +
>>   scripts/wic                                        | 13 +++----
>>   10 files changed, 68 insertions(+), 77 deletions(-)
>>

Re: [PATCH 0/6][dizzy] some wic bugfixes

[akuster808]

staged missing commit.

regards,
Armin

On 12/03/2014 05:40 AM, Philip Balister wrote:
> On 11/21/2014 11:07 AM, Tom Zanussi wrote:
>> >From the original posting:
>>
>> This patchset fixes a couple serious wic bugs:
>>
>> [YOCTO #6863] - Wic fails to create a working image for a large file system
>> [YOCTO #6290] - Update WIC for gummiboot
>>
>> I generated core-image-sdk images for each of the wic image types and
>> was able to fsck/mount the resulting filesystems.
>>
>> These should be applied to the next dizzy point release.
>>
>> The following changes since commit e69653b01dd8ad9fda2355b8112a65c813ce7fd5:
>>
>>    tzdata: update to 2014j (2014-11-17 08:01:21 -0800)
>>
>> are available in the git repository at:
>>
>>    git://git.yoctoproject.org/poky-contrib.git tzanussi/dizzy-next
>>    http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/dizzy-next
>>
>> Tom Zanussi (6):
>>    wic: Don't allow mkfs to fail silently in partition command
>>    wic: Use overhead factor when creating partitions from rootfs
>>      directories
>
> It looks like all the patches in this request made it into dizzy, except
> the overhead factor one. Is there a reason for this? Can we get the
> overhead one into dizzy also?
>
> Philip
>
>
>
>
>>    wic: Update the help text to include -D (--debug)
>>    Revert "wic: set bootimg_dir when using image-name artifacts"
>>    wic: Remove special-case bootimg_dir
>>    wic: Update bootimg-partition to use bootimg_dir
>>
>>   scripts/lib/image/engine.py                        | 42 +++++-----------------
>>   scripts/lib/image/help.py                          | 23 ++++++++----
>>   scripts/lib/wic/imager/direct.py                   |  5 +--
>>   .../lib/wic/kickstart/custom_commands/partition.py | 24 +++++++++----
>>   scripts/lib/wic/plugins/imager/direct_plugin.py    | 20 +++++------
>>   scripts/lib/wic/plugins/source/bootimg-efi.py      |  5 ++-
>>   .../lib/wic/plugins/source/bootimg-partition.py    |  9 +++--
>>   scripts/lib/wic/plugins/source/bootimg-pcbios.py   |  3 +-
>>   scripts/lib/wic/utils/oe/misc.py                   |  1 +
>>   scripts/wic                                        | 13 +++----
>>   10 files changed, 68 insertions(+), 77 deletions(-)
>>

Re: [PATCH 0/6][dizzy] some wic bugfixes

[Tom Zanussi]

On Wed, 2014-12-03 at 07:50 -0800, akuster808 wrote:
> 
> On 12/03/2014 05:40 AM, Philip Balister wrote:
> > On 11/21/2014 11:07 AM, Tom Zanussi wrote:
> >> >From the original posting:
> >>
> >> This patchset fixes a couple serious wic bugs:
> >>
> >> [YOCTO #6863] - Wic fails to create a working image for a large file system
> >> [YOCTO #6290] - Update WIC for gummiboot
> >>
> >> I generated core-image-sdk images for each of the wic image types and
> >> was able to fsck/mount the resulting filesystems.
> >>
> >> These should be applied to the next dizzy point release.
> >>
> >> The following changes since commit e69653b01dd8ad9fda2355b8112a65c813ce7fd5:
> >>
> >>    tzdata: update to 2014j (2014-11-17 08:01:21 -0800)
> >>
> >> are available in the git repository at:
> >>
> >>    git://git.yoctoproject.org/poky-contrib.git tzanussi/dizzy-next
> >>    http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/dizzy-next
> >>
> >> Tom Zanussi (6):
> >>    wic: Don't allow mkfs to fail silently in partition command
> >>    wic: Use overhead factor when creating partitions from rootfs
> >>      directories
> >
> > It looks like all the patches in this request made it into dizzy, except
> > the overhead factor one.
> 
> Then its not technically _all_ then ; )
> 
> Is there a reason for this?
> No reason. just missed it
> 
> Can we get the
> > overhead one into dizzy also?
> 
> sure.

And while the iron is hot, I'll ask about this one again:

  commit 4c222d3a67bae265ff42b448ef4a643b0131e578 
  Author: He Zhe <zhe.he@windriver.com>
  Date:   Tue Oct 21 17:47:44 2014 +0800

      kernel.bbclass: Create modules directory even if there is no modules
install


> 
> - Armin
> 
> >
> > Philip
> >
> >
> >
> >
> >>    wic: Update the help text to include -D (--debug)
> >>    Revert "wic: set bootimg_dir when using image-name artifacts"
> >>    wic: Remove special-case bootimg_dir
> >>    wic: Update bootimg-partition to use bootimg_dir
> >>
> >>   scripts/lib/image/engine.py                        | 42 +++++-----------------
> >>   scripts/lib/image/help.py                          | 23 ++++++++----
> >>   scripts/lib/wic/imager/direct.py                   |  5 +--
> >>   .../lib/wic/kickstart/custom_commands/partition.py | 24 +++++++++----
> >>   scripts/lib/wic/plugins/imager/direct_plugin.py    | 20 +++++------
> >>   scripts/lib/wic/plugins/source/bootimg-efi.py      |  5 ++-
> >>   .../lib/wic/plugins/source/bootimg-partition.py    |  9 +++--
> >>   scripts/lib/wic/plugins/source/bootimg-pcbios.py   |  3 +-
> >>   scripts/lib/wic/utils/oe/misc.py                   |  1 +
> >>   scripts/wic                                        | 13 +++----
> >>   10 files changed, 68 insertions(+), 77 deletions(-)
> >>