* [PATCH 1/9] wic-tools: add btrfs-tools squashfs-tools to DEPENDS
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
@ 2017-03-28 12:40 ` Ed Bartosh
2017-03-28 12:40 ` [PATCH 2/9] oe-selftest: add wic test case test_fs_types Ed Bartosh
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:40 UTC (permalink / raw)
To: openembedded-core
Added btrfs-tools-native and squashfs-tools-native to DEPENDS as
wic uses these tools to support btrfs and squashfs filesystems.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/recipes-core/meta/wic-tools.bb | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb
index bd4319a..cd494ec 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -2,7 +2,11 @@ SUMMARY = "A meta recipe to build native tools used by wic."
LICENSE = "MIT"
-DEPENDS = "parted-native syslinux-native gptfdisk-native dosfstools-native mtools-native bmap-tools-native grub-efi-native cdrtools-native"
+DEPENDS = "\
+ parted-native syslinux-native gptfdisk-native dosfstools-native \
+ mtools-native bmap-tools-native grub-efi-native cdrtools-native \
+ btrfs-tools-native squashfs-tools-native \
+ "
DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 2/9] oe-selftest: add wic test case test_fs_types
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
2017-03-28 12:40 ` [PATCH 1/9] wic-tools: add btrfs-tools squashfs-tools to DEPENDS Ed Bartosh
@ 2017-03-28 12:40 ` Ed Bartosh
2017-03-28 12:41 ` [PATCH 3/9] filemap: remove FilemapSeek class Ed Bartosh
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:40 UTC (permalink / raw)
To: openembedded-core
Added wic test case to test all possible filesystem
types for empty and not empty partitions.
[YOCTO #10618]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/lib/oeqa/selftest/wic.py | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index ee633f8..2af3bf5 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -717,3 +717,22 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
wksname = os.path.splitext(os.path.basename(wks.name))[0]
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+
+ def test_fs_types(self):
+ """Test filesystem types for empty and not empty partitions"""
+ img = 'core-image-minimal'
+ with NamedTemporaryFile("w", suffix=".wks") as wks:
+ wks.writelines(['part ext2 --fstype ext2 --source rootfs\n',
+ 'part btrfs --fstype btrfs --source rootfs --size 40M\n',
+ 'part squash --fstype squashfs --source rootfs\n',
+ 'part swap --fstype swap --size 1M\n',
+ 'part emptyvfat --fstype vfat --size 1M\n',
+ 'part emptyext2 --fstype ext2 --size 1M\n',
+ 'part emptybtrfs --fstype btrfs --size 100M\n',
+ 'part emptysquash --fstype squash --size 1M\n'])
+ wks.flush()
+ cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
+ self.assertEqual(0, runCmd(cmd).status)
+ wksname = os.path.splitext(os.path.basename(wks.name))[0]
+ out = glob(self.resultdir + "%s-*direct" % wksname)
+ self.assertEqual(1, len(out))
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 3/9] filemap: remove FilemapSeek class
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
2017-03-28 12:40 ` [PATCH 1/9] wic-tools: add btrfs-tools squashfs-tools to DEPENDS Ed Bartosh
2017-03-28 12:40 ` [PATCH 2/9] oe-selftest: add wic test case test_fs_types Ed Bartosh
@ 2017-03-28 12:41 ` Ed Bartosh
2017-03-28 12:41 ` [PATCH 4/9] oe-selftest: add kickstart_parser test case Ed Bartosh
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:41 UTC (permalink / raw)
To: openembedded-core
FIEMAP API was added to Linux kernel 2.6.28 back in 2008
SEEK_HOLE and SEEK_DATA API was added much letter.
As FIEMAP is used by filemap module as a default API it's
safe to remove FileMpSeek class as it's never used.
[YOCTO #10618]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
scripts/lib/wic/filemap.py | 309 +++++++--------------------------------------
1 file changed, 49 insertions(+), 260 deletions(-)
diff --git a/scripts/lib/wic/filemap.py b/scripts/lib/wic/filemap.py
index 080668e..34523c0 100644
--- a/scripts/lib/wic/filemap.py
+++ b/scripts/lib/wic/filemap.py
@@ -54,24 +54,46 @@ class Error(Exception):
"""A class for all the other exceptions raised by this module."""
pass
+# Below goes the FIEMAP ioctl implementation, which is not very readable
+# because it deals with the rather complex FIEMAP ioctl. To understand the
+# code, you need to know the FIEMAP interface, which is documented in the
+# "Documentation/filesystems/fiemap.txt" file in the Linux kernel sources.
+
+# Format string for 'struct fiemap'
+_FIEMAP_FORMAT = "=QQLLLL"
+# sizeof(struct fiemap)
+_FIEMAP_SIZE = struct.calcsize(_FIEMAP_FORMAT)
+# Format string for 'struct fiemap_extent'
+_FIEMAP_EXTENT_FORMAT = "=QQQQQLLLL"
+# sizeof(struct fiemap_extent)
+_FIEMAP_EXTENT_SIZE = struct.calcsize(_FIEMAP_EXTENT_FORMAT)
+# The FIEMAP ioctl number
+_FIEMAP_IOCTL = 0xC020660B
+# This FIEMAP ioctl flag which instructs the kernel to sync the file before
+# reading the block map
+_FIEMAP_FLAG_SYNC = 0x00000001
+# Size of the buffer for 'struct fiemap_extent' elements which will be used
+# when invoking the FIEMAP ioctl. The larger is the buffer, the less times the
+# FIEMAP ioctl will be invoked.
+_FIEMAP_BUFFER_SIZE = 256 * 1024
-class _FilemapBase(object):
+class FilemapFiemap:
"""
- This is a base class for a couple of other classes in this module. This
- class simply performs the common parts of the initialization process: opens
- the image file, gets its size, etc. The 'log' parameter is the logger object
- to use for printing messages.
+ This class provides API to the FIEMAP ioctl. Namely, it allows to iterate
+ over all mapped blocks and over all holes.
+
+ This class synchronizes the image file every time it invokes the FIEMAP
+ ioctl in order to work-around early FIEMAP implementation kernel bugs.
"""
- def __init__(self, image, log=None):
+ def __init__(self, image):
"""
- Initialize a class instance. The 'image' argument is full path to the
- file or file object to operate on.
+ Initialize a class instance. The 'image' argument is full the file
+ object to operate on.
"""
+ self._log = logging.getLogger(__name__)
- self._log = log
- if self._log is None:
- self._log = logging.getLogger(__name__)
+ self._log.debug("FilemapFiemap: initializing")
self._f_image_needs_close = False
@@ -113,240 +135,6 @@ class _FilemapBase(object):
self._log.debug("block size %d, blocks count %d, image size %d"
% (self.block_size, self.blocks_cnt, self.image_size))
- def __del__(self):
- """The class destructor which just closes the image file."""
- if self._f_image_needs_close:
- self._f_image.close()
-
- def _open_image_file(self):
- """Open the image file."""
- try:
- self._f_image = open(self._image_path, 'rb')
- except IOError as err:
- raise Error("cannot open image file '%s': %s"
- % (self._image_path, err))
-
- self._f_image_needs_close = True
-
- def block_is_mapped(self, block): # pylint: disable=W0613,R0201
- """
- This method has has to be implemented by child classes. It returns
- 'True' if block number 'block' of the image file is mapped and 'False'
- otherwise.
- """
-
- raise Error("the method is not implemented")
-
- def block_is_unmapped(self, block): # pylint: disable=W0613,R0201
- """
- This method has has to be implemented by child classes. It returns
- 'True' if block number 'block' of the image file is not mapped (hole)
- and 'False' otherwise.
- """
-
- raise Error("the method is not implemented")
-
- def get_mapped_ranges(self, start, count): # pylint: disable=W0613,R0201
- """
- This method has has to be implemented by child classes. This is a
- generator which yields ranges of mapped blocks in the file. The ranges
- are tuples of 2 elements: [first, last], where 'first' is the first
- mapped block and 'last' is the last mapped block.
-
- The ranges are yielded for the area of the file of size 'count' blocks,
- starting from block 'start'.
- """
-
- raise Error("the method is not implemented")
-
- def get_unmapped_ranges(self, start, count): # pylint: disable=W0613,R0201
- """
- This method has has to be implemented by child classes. Just like
- 'get_mapped_ranges()', but yields unmapped block ranges instead
- (holes).
- """
-
- raise Error("the method is not implemented")
-
-
-# The 'SEEK_HOLE' and 'SEEK_DATA' options of the file seek system call
-_SEEK_DATA = 3
-_SEEK_HOLE = 4
-
-def _lseek(file_obj, offset, whence):
- """This is a helper function which invokes 'os.lseek' for file object
- 'file_obj' and with specified 'offset' and 'whence'. The 'whence'
- argument is supposed to be either '_SEEK_DATA' or '_SEEK_HOLE'. When
- there is no more data or hole starting from 'offset', this function
- returns '-1'. Otherwise the data or hole position is returned."""
-
- try:
- return os.lseek(file_obj.fileno(), offset, whence)
- except OSError as err:
- # The 'lseek' system call returns the ENXIO if there is no data or
- # hole starting from the specified offset.
- if err.errno == os.errno.ENXIO:
- return -1
- elif err.errno == os.errno.EINVAL:
- raise ErrorNotSupp("the kernel or file-system does not support "
- "\"SEEK_HOLE\" and \"SEEK_DATA\"")
- else:
- raise
-
-class FilemapSeek(_FilemapBase):
- """
- This class uses the 'SEEK_HOLE' and 'SEEK_DATA' to find file block mapping.
- Unfortunately, the current implementation requires the caller to have write
- access to the image file.
- """
-
- def __init__(self, image, log=None):
- """Refer the '_FilemapBase' class for the documentation."""
-
- # Call the base class constructor first
- _FilemapBase.__init__(self, image, log)
- self._log.debug("FilemapSeek: initializing")
-
- self._probe_seek_hole()
-
- def _probe_seek_hole(self):
- """
- Check whether the system implements 'SEEK_HOLE' and 'SEEK_DATA'.
- Unfortunately, there seems to be no clean way for detecting this,
- because often the system just fakes them by just assuming that all
- files are fully mapped, so 'SEEK_HOLE' always returns EOF and
- 'SEEK_DATA' always returns the requested offset.
-
- I could not invent a better way of detecting the fake 'SEEK_HOLE'
- implementation than just to create a temporary file in the same
- directory where the image file resides. It would be nice to change this
- to something better.
- """
-
- directory = os.path.dirname(self._image_path)
-
- try:
- tmp_obj = tempfile.TemporaryFile("w+", dir=directory)
- except IOError as err:
- raise ErrorNotSupp("cannot create a temporary in \"%s\": %s"
- % (directory, err))
-
- try:
- os.ftruncate(tmp_obj.fileno(), self.block_size)
- except OSError as err:
- raise ErrorNotSupp("cannot truncate temporary file in \"%s\": %s"
- % (directory, err))
-
- offs = _lseek(tmp_obj, 0, _SEEK_HOLE)
- if offs != 0:
- # We are dealing with the stub 'SEEK_HOLE' implementation which
- # always returns EOF.
- self._log.debug("lseek(0, SEEK_HOLE) returned %d" % offs)
- raise ErrorNotSupp("the file-system does not support "
- "\"SEEK_HOLE\" and \"SEEK_DATA\" but only "
- "provides a stub implementation")
-
- tmp_obj.close()
-
- def block_is_mapped(self, block):
- """Refer the '_FilemapBase' class for the documentation."""
- offs = _lseek(self._f_image, block * self.block_size, _SEEK_DATA)
- if offs == -1:
- result = False
- else:
- result = (offs // self.block_size == block)
-
- self._log.debug("FilemapSeek: block_is_mapped(%d) returns %s"
- % (block, result))
- return result
-
- def block_is_unmapped(self, block):
- """Refer the '_FilemapBase' class for the documentation."""
- return not self.block_is_mapped(block)
-
- def _get_ranges(self, start, count, whence1, whence2):
- """
- This function implements 'get_mapped_ranges()' and
- 'get_unmapped_ranges()' depending on what is passed in the 'whence1'
- and 'whence2' arguments.
- """
-
- assert whence1 != whence2
- end = start * self.block_size
- limit = end + count * self.block_size
-
- while True:
- start = _lseek(self._f_image, end, whence1)
- if start == -1 or start >= limit or start == self.image_size:
- break
-
- end = _lseek(self._f_image, start, whence2)
- if end == -1 or end == self.image_size:
- end = self.blocks_cnt * self.block_size
- if end > limit:
- end = limit
-
- start_blk = start // self.block_size
- end_blk = end // self.block_size - 1
- self._log.debug("FilemapSeek: yielding range (%d, %d)"
- % (start_blk, end_blk))
- yield (start_blk, end_blk)
-
- def get_mapped_ranges(self, start, count):
- """Refer the '_FilemapBase' class for the documentation."""
- self._log.debug("FilemapSeek: get_mapped_ranges(%d, %d(%d))"
- % (start, count, start + count - 1))
- return self._get_ranges(start, count, _SEEK_DATA, _SEEK_HOLE)
-
- def get_unmapped_ranges(self, start, count):
- """Refer the '_FilemapBase' class for the documentation."""
- self._log.debug("FilemapSeek: get_unmapped_ranges(%d, %d(%d))"
- % (start, count, start + count - 1))
- return self._get_ranges(start, count, _SEEK_HOLE, _SEEK_DATA)
-
-
-# Below goes the FIEMAP ioctl implementation, which is not very readable
-# because it deals with the rather complex FIEMAP ioctl. To understand the
-# code, you need to know the FIEMAP interface, which is documented in the
-# "Documentation/filesystems/fiemap.txt" file in the Linux kernel sources.
-
-# Format string for 'struct fiemap'
-_FIEMAP_FORMAT = "=QQLLLL"
-# sizeof(struct fiemap)
-_FIEMAP_SIZE = struct.calcsize(_FIEMAP_FORMAT)
-# Format string for 'struct fiemap_extent'
-_FIEMAP_EXTENT_FORMAT = "=QQQQQLLLL"
-# sizeof(struct fiemap_extent)
-_FIEMAP_EXTENT_SIZE = struct.calcsize(_FIEMAP_EXTENT_FORMAT)
-# The FIEMAP ioctl number
-_FIEMAP_IOCTL = 0xC020660B
-# This FIEMAP ioctl flag which instructs the kernel to sync the file before
-# reading the block map
-_FIEMAP_FLAG_SYNC = 0x00000001
-# Size of the buffer for 'struct fiemap_extent' elements which will be used
-# when invoking the FIEMAP ioctl. The larger is the buffer, the less times the
-# FIEMAP ioctl will be invoked.
-_FIEMAP_BUFFER_SIZE = 256 * 1024
-
-class FilemapFiemap(_FilemapBase):
- """
- This class provides API to the FIEMAP ioctl. Namely, it allows to iterate
- over all mapped blocks and over all holes.
-
- This class synchronizes the image file every time it invokes the FIEMAP
- ioctl in order to work-around early FIEMAP implementation kernel bugs.
- """
-
- def __init__(self, image, log=None):
- """
- Initialize a class instance. The 'image' argument is full the file
- object to operate on.
- """
-
- # Call the base class constructor first
- _FilemapBase.__init__(self, image, log)
- self._log.debug("FilemapFiemap: initializing")
-
self._buf_size = _FIEMAP_BUFFER_SIZE
# Calculate how many 'struct fiemap_extent' elements fit the buffer
@@ -362,6 +150,21 @@ class FilemapFiemap(_FilemapBase):
# Check if the FIEMAP ioctl is supported
self.block_is_mapped(0)
+ def __del__(self):
+ """The class destructor which just closes the image file."""
+ if self._f_image_needs_close:
+ self._f_image.close()
+
+ def _open_image_file(self):
+ """Open the image file."""
+ try:
+ self._f_image = open(self._image_path, 'rb')
+ except IOError as err:
+ raise Error("cannot open image file '%s': %s"
+ % (self._image_path, err))
+
+ self._f_image_needs_close = True
+
def _invoke_fiemap(self, block, count):
"""
Invoke the FIEMAP ioctl for 'count' blocks of the file starting from
@@ -515,24 +318,10 @@ class FilemapFiemap(_FilemapBase):
% (hole_first, start + count - 1))
yield (hole_first, start + count - 1)
-def filemap(image, log=None):
- """
- Create and return an instance of a Filemap class - 'FilemapFiemap' or
- 'FilemapSeek', depending on what the system we run on supports. If the
- FIEMAP ioctl is supported, an instance of the 'FilemapFiemap' class is
- returned. Otherwise, if 'SEEK_HOLE' is supported an instance of the
- 'FilemapSeek' class is returned. If none of these are supported, the
- function generates an 'Error' type exception.
- """
-
- try:
- return FilemapFiemap(image, log)
- except ErrorNotSupp:
- return FilemapSeek(image, log)
def sparse_copy(src_fname, dst_fname, offset=0, skip=0):
"""Efficiently copy sparse file to or into another file."""
- fmap = filemap(src_fname)
+ fmap = FilemapFiemap(src_fname)
try:
dst_file = open(dst_fname, 'r+b')
except IOError:
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 4/9] oe-selftest: add kickstart_parser test case
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
` (2 preceding siblings ...)
2017-03-28 12:41 ` [PATCH 3/9] filemap: remove FilemapSeek class Ed Bartosh
@ 2017-03-28 12:41 ` Ed Bartosh
2017-03-28 12:41 ` [PATCH 5/9] wic: remove fsimage plugin Ed Bartosh
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:41 UTC (permalink / raw)
To: openembedded-core
Added test_kickstart_parser test case to test wks parser
options not yet covered by tests.
[YOCTO #10618]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/lib/oeqa/selftest/wic.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 2af3bf5..f0d9dd9 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -736,3 +736,15 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
wksname = os.path.splitext(os.path.basename(wks.name))[0]
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+
+ def test_kickstart_parser(self):
+ """Test wks parser options"""
+ with NamedTemporaryFile("w", suffix=".wks") as wks:
+ wks.writelines(['part / --fstype ext3 --source rootfs --system-id 0xFF '\
+ '--overhead-factor 1.2 --size 100k\n'])
+ wks.flush()
+ cmd = "wic create %s -e core-image-minimal -o %s" % (wks.name, self.resultdir)
+ self.assertEqual(0, runCmd(cmd).status)
+ wksname = os.path.splitext(os.path.basename(wks.name))[0]
+ out = glob(self.resultdir + "%s-*direct" % wksname)
+ self.assertEqual(1, len(out))
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 5/9] wic: remove fsimage plugin
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
` (3 preceding siblings ...)
2017-03-28 12:41 ` [PATCH 4/9] oe-selftest: add kickstart_parser test case Ed Bartosh
@ 2017-03-28 12:41 ` Ed Bartosh
2017-03-28 12:41 ` [PATCH 6/9] wic: use wic-tools STAGING_DATADIR as bootimg_dir Ed Bartosh
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:41 UTC (permalink / raw)
To: openembedded-core
Removed fsimage plugin and prepare_rootfs_from_fs_image API as
they duplicate functionality of rawcopy plugin. Fsimage plugin makes
wic to remove the image artifact from deployment directory, which
can cause problems too.
[YOCTO #10618]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
scripts/lib/wic/partition.py | 13 -------
scripts/lib/wic/plugins/source/fsimage.py | 56 -------------------------------
2 files changed, 69 deletions(-)
delete mode 100644 scripts/lib/wic/plugins/source/fsimage.py
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 8e32afc..647a6fb 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -193,19 +193,6 @@ class Partition():
"larger (%d kB) than its allowed size %d kB" %
(self.mountpoint, self.size, self.fixed_size))
- def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir,
- rootfs_dir):
- """
- Handle an already-created partition e.g. xxx.ext3
- """
- rootfs = oe_builddir
- du_cmd = "du -Lbks %s" % rootfs
- out = exec_cmd(du_cmd)
- rootfs_size = out.split()[0]
-
- self.size = int(rootfs_size)
- self.source_file = rootfs
-
def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
native_sysroot):
"""
diff --git a/scripts/lib/wic/plugins/source/fsimage.py b/scripts/lib/wic/plugins/source/fsimage.py
deleted file mode 100644
index f781499..0000000
--- a/scripts/lib/wic/plugins/source/fsimage.py
+++ /dev/null
@@ -1,56 +0,0 @@
-# ex:ts=4:sw=4:sts=4:et
-# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 2 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License along
-# with this program; if not, write to the Free Software Foundation, Inc.,
-# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
-#
-
-import logging
-import os
-
-from wic import WicError
-from wic.pluginbase import SourcePlugin
-from wic.utils.misc import get_bitbake_var
-
-logger = logging.getLogger('wic')
-
-class FSImagePlugin(SourcePlugin):
- """
- Add an already existing filesystem image to the partition layout.
- """
-
- name = 'fsimage'
-
- @classmethod
- def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
- oe_builddir, bootimg_dir, kernel_dir,
- rootfs_dir, native_sysroot):
- """
- Called to do the actual content population for a partition i.e. it
- 'prepares' the partition to be incorporated into the image.
- """
- if not bootimg_dir:
- bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
- if not bootimg_dir:
- raise WicError("Couldn't find DEPLOY_DIR_IMAGE, exiting")
-
- logger.debug('Bootimg dir: %s', bootimg_dir)
-
- if 'file' not in source_params:
- raise WicError("No file specified")
-
- src = os.path.join(bootimg_dir, source_params['file'])
-
-
- logger.debug('Preparing partition using image %s', src)
- part.prepare_rootfs_from_fs_image(cr_workdir, src, "")
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 6/9] wic: use wic-tools STAGING_DATADIR as bootimg_dir
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
` (4 preceding siblings ...)
2017-03-28 12:41 ` [PATCH 5/9] wic: remove fsimage plugin Ed Bartosh
@ 2017-03-28 12:41 ` Ed Bartosh
2017-03-28 12:41 ` [PATCH 7/9] wic: remove runner.show API Ed Bartosh
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:41 UTC (permalink / raw)
To: openembedded-core
If bootloader artifacts are not found in default bootimg_dir
use wic-tools sysroot for the same purpose. This should
prevent wic from failing if bootloader artifacts can't be
found in image native sysroot.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
scripts/lib/wic/plugins/source/bootimg-pcbios.py | 42 +++++++++++-------------
1 file changed, 20 insertions(+), 22 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index 11db304..bfb8ff5 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -44,19 +44,16 @@ class BootimgPcbiosPlugin(SourcePlugin):
name = 'bootimg-pcbios'
@classmethod
- def _get_syslinux_dir(cls, bootimg_dir):
+ def _get_bootimg_dir(cls, bootimg_dir, dirname):
"""
- Get path to syslinux from either default bootimg_dir
- or wic-tools STAGING_DIR.
+ Check if dirname exists in default bootimg_dir or
+ in wic-tools STAGING_DIR.
"""
- for path in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
- if not path:
- continue
- syslinux_dir = os.path.join(path, 'syslinux')
- if os.path.exists(syslinux_dir):
- return syslinux_dir
+ for result in (bootimg_dir, get_bitbake_var("STAGING_DATADIR", "wic-tools")):
+ if os.path.exists("%s/%s" % (result, dirname)):
+ return result
- raise WicError("Couldn't find syslinux directory, exiting")
+ raise WicError("Couldn't find correct bootimg_dir, exiting")
@classmethod
def do_install_disk(cls, disk, disk_name, creator, workdir, oe_builddir,
@@ -65,11 +62,12 @@ class BootimgPcbiosPlugin(SourcePlugin):
Called after all partitions have been prepared and assembled into a
disk image. In this case, we install the MBR.
"""
- syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
+ bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
+ mbrfile = "%s/syslinux/" % bootimg_dir
if creator.ptable_format == 'msdos':
- mbrfile = os.path.join(syslinux_dir, "mbr.bin")
+ mbrfile += "mbr.bin"
elif creator.ptable_format == 'gpt':
- mbrfile = os.path.join(syslinux_dir, "gptmbr.bin")
+ mbrfile += "gptmbr.bin"
else:
raise WicError("Unsupported partition table: %s" %
creator.ptable_format)
@@ -155,7 +153,7 @@ class BootimgPcbiosPlugin(SourcePlugin):
'prepares' the partition to be incorporated into the image.
In this case, prepare content for legacy bios boot partition.
"""
- syslinux_dir = cls._get_syslinux_dir(bootimg_dir)
+ bootimg_dir = cls._get_bootimg_dir(bootimg_dir, 'syslinux')
staging_kernel_dir = kernel_dir
@@ -163,14 +161,14 @@ class BootimgPcbiosPlugin(SourcePlugin):
cmds = ("install -m 0644 %s/bzImage %s/vmlinuz" %
(staging_kernel_dir, hdddir),
- "install -m 444 %s/ldlinux.sys %s/ldlinux.sys" %
- (syslinux_dir, hdddir),
- "install -m 0644 %s/vesamenu.c32 %s/vesamenu.c32" %
- (syslinux_dir, hdddir),
- "install -m 444 %s/libcom32.c32 %s/libcom32.c32" %
- (syslinux_dir, hdddir),
- "install -m 444 %s/libutil.c32 %s/libutil.c32" %
- (syslinux_dir, hdddir))
+ "install -m 444 %s/syslinux/ldlinux.sys %s/ldlinux.sys" %
+ (bootimg_dir, hdddir),
+ "install -m 0644 %s/syslinux/vesamenu.c32 %s/vesamenu.c32" %
+ (bootimg_dir, hdddir),
+ "install -m 444 %s/syslinux/libcom32.c32 %s/libcom32.c32" %
+ (bootimg_dir, hdddir),
+ "install -m 444 %s/syslinux/libutil.c32 %s/libutil.c32" %
+ (bootimg_dir, hdddir))
for install_cmd in cmds:
exec_cmd(install_cmd)
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 7/9] wic: remove runner.show API
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
` (5 preceding siblings ...)
2017-03-28 12:41 ` [PATCH 6/9] wic: use wic-tools STAGING_DATADIR as bootimg_dir Ed Bartosh
@ 2017-03-28 12:41 ` Ed Bartosh
2017-03-28 12:41 ` [PATCH 8/9] wic: remove unused code from runner module Ed Bartosh
2017-03-28 12:41 ` [PATCH 9/9] oe-selftest: add test_image_bootpart_globbed test for wic Ed Bartosh
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:41 UTC (permalink / raw)
To: openembedded-core
Replaced runner.show call to exec_cmd call in bootimg-pcbios
plugin. Removed runner.show API as it's not used anywhere else.
[YOCTO #10618]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
scripts/lib/wic/plugins/source/bootimg-pcbios.py | 6 ++----
scripts/lib/wic/utils/runner.py | 24 ------------------------
2 files changed, 2 insertions(+), 28 deletions(-)
diff --git a/scripts/lib/wic/plugins/source/bootimg-pcbios.py b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
index bfb8ff5..5890c12 100644
--- a/scripts/lib/wic/plugins/source/bootimg-pcbios.py
+++ b/scripts/lib/wic/plugins/source/bootimg-pcbios.py
@@ -81,10 +81,8 @@ class BootimgPcbiosPlugin(SourcePlugin):
logger.debug("Installing MBR on disk %s as %s with size %s bytes",
disk_name, full_path, disk.min_size)
- rcode = runner.show(['dd', 'if=%s' % mbrfile,
- 'of=%s' % full_path, 'conv=notrunc'])
- if rcode != 0:
- raise WicError("Unable to set MBR to %s" % full_path)
+ dd_cmd = "dd if=%s of=%s conv=notrunc" % (mbrfile, full_path)
+ exec_cmd(dd_cmd, native_sysroot)
@classmethod
def do_configure_partition(cls, part, source_params, creator, cr_workdir,
diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py
index 56d7ea3..348557a 100644
--- a/scripts/lib/wic/utils/runner.py
+++ b/scripts/lib/wic/utils/runner.py
@@ -82,30 +82,6 @@ def runtool(cmdln_or_args, catch=1):
return (process.returncode, out)
-def show(cmdln_or_args):
- """Show all messages using logger.debug."""
-
- rcode, out = runtool(cmdln_or_args, catch=3)
-
- if isinstance(cmdln_or_args, list):
- cmd = ' '.join(cmdln_or_args)
- else:
- cmd = cmdln_or_args
-
- msg = 'running command: "%s"' % cmd
- if out:
- out = out.strip()
- if out:
- msg += ', with output::'
- msg += '\n +----------------'
- for line in out.splitlines():
- msg += '\n | %s' % line
- msg += '\n +----------------'
-
- logger.debug(msg)
-
- return rcode
-
def outs(cmdln_or_args, catch=1):
# get the outputs of tools
return runtool(cmdln_or_args, catch)[1].strip()
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 8/9] wic: remove unused code from runner module
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
` (6 preceding siblings ...)
2017-03-28 12:41 ` [PATCH 7/9] wic: remove runner.show API Ed Bartosh
@ 2017-03-28 12:41 ` Ed Bartosh
2017-03-28 12:41 ` [PATCH 9/9] oe-selftest: add test_image_bootpart_globbed test for wic Ed Bartosh
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:41 UTC (permalink / raw)
To: openembedded-core
Removed unused APIs 'outs' and 'quiet'.
Removed 'catch' parameter from runner.runtool API as wic
uses only one value of it. Removed the code that handles
unused values of 'catch' parameter.
[YOCTO #10618]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
scripts/lib/wic/utils/misc.py | 14 ++++++------
scripts/lib/wic/utils/runner.py | 50 +++++------------------------------------
2 files changed, 13 insertions(+), 51 deletions(-)
diff --git a/scripts/lib/wic/utils/misc.py b/scripts/lib/wic/utils/misc.py
index c941112..307779a 100644
--- a/scripts/lib/wic/utils/misc.py
+++ b/scripts/lib/wic/utils/misc.py
@@ -59,7 +59,7 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
"syslinux": "syslinux"
}
-def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
+def _exec_cmd(cmd_and_args, as_shell=False):
"""
Execute command, catching stderr, stdout
@@ -70,9 +70,9 @@ def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
logger.debug(args)
if as_shell:
- ret, out = runner.runtool(cmd_and_args, catch)
+ ret, out = runner.runtool(cmd_and_args)
else:
- ret, out = runner.runtool(args, catch)
+ ret, out = runner.runtool(args)
out = out.strip()
if ret != 0:
raise WicError("_exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
@@ -84,14 +84,14 @@ def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
return ret, out
-def exec_cmd(cmd_and_args, as_shell=False, catch=3):
+def exec_cmd(cmd_and_args, as_shell=False):
"""
Execute command, return output
"""
- return _exec_cmd(cmd_and_args, as_shell, catch)[1]
+ return _exec_cmd(cmd_and_args, as_shell)[1]
-def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
+def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
"""
Execute native command, catching stderr, stdout
@@ -118,7 +118,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3, pseudo=""):
# If the command isn't in the native sysroot say we failed.
if spawn.find_executable(args[0], native_paths):
- ret, out = _exec_cmd(native_cmd_and_args, True, catch)
+ ret, out = _exec_cmd(native_cmd_and_args, True)
else:
ret = 127
out = "can't find native executable %s in %s" % (args[0], native_paths)
diff --git a/scripts/lib/wic/utils/runner.py b/scripts/lib/wic/utils/runner.py
index 348557a..4aa00fb 100644
--- a/scripts/lib/wic/utils/runner.py
+++ b/scripts/lib/wic/utils/runner.py
@@ -14,32 +14,17 @@
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc., 59
# Temple Place - Suite 330, Boston, MA 02111-1307, USA.
-
-import logging
-import os
import subprocess
from wic import WicError
-logger = logging.getLogger('wic')
-
-def runtool(cmdln_or_args, catch=1):
+def runtool(cmdln_or_args):
""" wrapper for most of the subprocess calls
input:
cmdln_or_args: can be both args and cmdln str (shell=True)
- catch: 0, quitely run
- 1, only STDOUT
- 2, only STDERR
- 3, both STDOUT and STDERR
return:
- (rc, output)
- if catch==0: the output will always None
+ rc, output
"""
-
- if catch not in (0, 1, 2, 3):
- # invalid catch selection, will cause exception, that's good
- return None
-
if isinstance(cmdln_or_args, list):
cmd = cmdln_or_args[0]
shell = False
@@ -48,26 +33,13 @@ def runtool(cmdln_or_args, catch=1):
cmd = shlex.split(cmdln_or_args)[0]
shell = True
- if catch != 3:
- dev_null = os.open("/dev/null", os.O_WRONLY)
-
- if catch == 0:
- sout = dev_null
- serr = dev_null
- elif catch == 1:
- sout = subprocess.PIPE
- serr = dev_null
- elif catch == 2:
- sout = dev_null
- serr = subprocess.PIPE
- elif catch == 3:
- sout = subprocess.PIPE
- serr = subprocess.STDOUT
+ sout = subprocess.PIPE
+ serr = subprocess.STDOUT
try:
process = subprocess.Popen(cmdln_or_args, stdout=sout,
stderr=serr, shell=shell)
- (sout, serr) = process.communicate()
+ sout, serr = process.communicate()
# combine stdout and stderr, filter None out and decode
out = ''.join([out.decode('utf-8') for out in [sout, serr] if out])
except OSError as err:
@@ -76,15 +48,5 @@ def runtool(cmdln_or_args, catch=1):
raise WicError('Cannot run command: %s, lost dependency?' % cmd)
else:
raise # relay
- finally:
- if catch != 3:
- os.close(dev_null)
-
- return (process.returncode, out)
-
-def outs(cmdln_or_args, catch=1):
- # get the outputs of tools
- return runtool(cmdln_or_args, catch)[1].strip()
-def quiet(cmdln_or_args):
- return runtool(cmdln_or_args, catch=0)[0]
+ return process.returncode, out
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 9/9] oe-selftest: add test_image_bootpart_globbed test for wic
2017-03-28 12:40 [PATCH 0/9] #10618: Increase oe-selftest coverage of wic codebase (last part) Ed Bartosh
` (7 preceding siblings ...)
2017-03-28 12:41 ` [PATCH 8/9] wic: remove unused code from runner module Ed Bartosh
@ 2017-03-28 12:41 ` Ed Bartosh
8 siblings, 0 replies; 10+ messages in thread
From: Ed Bartosh @ 2017-03-28 12:41 UTC (permalink / raw)
To: openembedded-core
Test image-bootpart wic plugin with globbed value of
IMAGE_BOOT_FILES variable to increase test coverage.
[YOCTO #10618]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
meta/lib/oeqa/selftest/wic.py | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index f0d9dd9..2401aaa 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -748,3 +748,13 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
wksname = os.path.splitext(os.path.basename(wks.name))[0]
out = glob(self.resultdir + "%s-*direct" % wksname)
self.assertEqual(1, len(out))
+
+ def test_image_bootpart_globbed(self):
+ """Test globbed sources with image-bootpart plugin"""
+ img = "core-image-minimal"
+ cmd = "wic create sdimage-bootpart -e %s -o %s" % (img, self.resultdir)
+ config = 'IMAGE_BOOT_FILES = "%s*"' % get_bb_var('KERNEL_IMAGETYPE', img)
+ self.append_config(config)
+ self.assertEqual(0, runCmd(cmd).status)
+ self.remove_config(config)
+ self.assertEqual(1, len(glob(self.resultdir + "sdimage-bootpart-*direct")))
--
2.1.4
^ permalink raw reply related [flat|nested] 10+ messages in thread