[PATCH v7 00/10] Fix permissions and embed-rotofs

[PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Today wic behaves differently if we run it from bitbake of directly from the
commandline.

When it is run from bitbake, the permissions/usersnames are handled by the
pseudo database of the main image.

When it is run from the comandline ,it is run outside the main image database.

This results on permissions/usernames not working ok on some usecases on both
bitbake and wic.


This is an attempt to fix all the permission bugs that I am aware from wic.
Using the following usecases:


#exclude-path
part / --source rootfs --fstype=ext4 --exclude-path=home

#split_partition
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/ --fstype=ext4

#multi_partition
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4


With the current master:
-------------------------
#exclude-path
From Bitbake: OK
wic cmdline: FAIL

#split_partition
From Bitbake: OK
wic cmdline: FAIL, permissions invalid on both partitions

#multi_partition
From Bitbake: FAIL second partition
wic cmdline: OK


After:  wic: Fix permissions when using exclude or include path
--------------------------------------------------------------------
#exclude-path
From Bitbake: OK
wic cmdline: OK

#split_partition
From Bitbake: OK
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
From Bitbake: FAIL second partition
wic cmdline: OK


After: wic: Fix multi images .wks with bitbake
----------------------------------------------
#exclude-path
From Bitbake: OK
wic cmdline: OK

#split_partition
From Bitbake: FAIL, permissions invalid on second partition
wic cmdline: FAIL, permissions invalid on second partition

#multi_partition
From Bitbake: OK
wic cmdline: OK



After: wic: Add --change-directory argument
--------------------------------------------

we can have a .wks like:
part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
part /etc --source rootfs --fstype=ext4 --change-directory=etc


Wich works fine from bitbake and from cmdline, and has the same functionality as
split_partition.


Changelog v6->v7 (Thanks Paul):
  - Use passwd instead of shadow for oetests
  - For pseudo folder inside workdir
  - Improve comments on tar
  - Use line number on tar name
  - Allow include-files of files in workdir

Changelog v5->v6:
  - Allow two arguments to include-path
  - Fix permission on include-path
  - exec_native with ASSUME_PROVIDED
  - More unittesting

Changelog v4->v5:
  - Add unittesting
  - wic: Avoid creating invalid pseudo directory

Ricardo Ribalda Delgado (10):
  wic: Fix permissions when using exclude or include path
  wic: Fix multi images .wks with bitbake
  wic: Add --change-directory argument
  wic: Continue if excluded_path does not exist
  wic: Avoid creating invalid pseudo directory
  oeqa: wic: Add tests for permissions and change-directory
  wic: misc: Do not find for executables in ALREADY_PROVIDED
  wic: root: Add an opt. destination on include-path
  wic: rootfs: Combine path_validation in one function
  oeqa: wic: Add more tests for include_path

 meta/classes/image_types_wic.bbclass     |   8 +-
 meta/lib/oeqa/selftest/cases/wic.py      | 153 ++++++++++++++++++++++-
 meta/recipes-core/meta/wic-tools.bb      |   2 +-
 scripts/lib/wic/help.py                  |  26 ++--
 scripts/lib/wic/ksparser.py              |   3 +-
 scripts/lib/wic/misc.py                  |  14 ++-
 scripts/lib/wic/partition.py             |  20 +--
 scripts/lib/wic/plugins/source/rootfs.py | 133 +++++++++++++++++---
 8 files changed, 318 insertions(+), 41 deletions(-)

-- 
2.25.1

[PATCH v7 01/10] wic: Fix permissions when using exclude or include path

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

When parameters include_path or exclude_path are passed to the rootfs
plugin, it will copy the partition content into a folder and make all
the modifications there.

This is done using copyhardlinktree(), which does not take into
consideration the content of the pseudo folder, which contains the
information about the right permissions and ownership of the folders.

This results in a rootfs owned by the user that is running the wic
command (usually UID 1000), which makes some rootfs unbootable.

This bug can be easily triggerd with the following .wks

part / --source rootfs --fstype=ext4 --exclude-path=home

And this sequence:

$ wic create test-permissions -e core-image-minimal -o test/
$ sudo mount test/test-permissions-202004080823-sda.direct.p1 /mnt
$ ls -la /mnt/etc/shadow

To fix this we copy the content of the pseudo folders to the new folder
and modify the pseudo database using the "pseudo -B" command.

If the rootfs is not a rootfs generated by bitbake a warning is shown
making the user aware that the permissions on the target might not match
what he expects.

WARNING: /tmp/test/../pseudo folder does not exist. Usernames and permissions will be invalid

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/partition.py             |  7 +++--
 scripts/lib/wic/plugins/source/rootfs.py | 37 ++++++++++++++++++++++--
 2 files changed, 38 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 2d95f78439..b02711be37 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -190,7 +190,7 @@ class Partition():
                            (self.mountpoint, self.size, self.fixed_size))
 
     def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
-                       native_sysroot, real_rootfs = True):
+                       native_sysroot, real_rootfs = True, pseudo_dir = None):
         """
         Prepare content for a rootfs partition i.e. create a partition
         and fill it from a /rootfs dir.
@@ -198,8 +198,9 @@ class Partition():
         Currently handles ext2/3/4, btrfs, vfat and squashfs.
         """
         p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR",
-                                         "%s/../pseudo" %  rootfs_dir)
+        if (pseudo_dir == None):
+            pseudo_dir = "%s/../pseudo" %  rootfs_dir
+        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
         p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
         p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
         pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 705aeb5563..8b2a067385 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -20,7 +20,7 @@ from oe.path import copyhardlinktree
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
-from wic.misc import get_bitbake_var
+from wic.misc import get_bitbake_var, exec_native_cmd
 
 logger = logging.getLogger('wic')
 
@@ -44,6 +44,15 @@ class RootfsPlugin(SourcePlugin):
 
         return os.path.realpath(image_rootfs_dir)
 
+    @staticmethod
+    def __get_pseudo(native_sysroot, rootfs, pseudo_dir):
+        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs
+        pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+        pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+        return pseudo
+
     @classmethod
     def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
                              oe_builddir, bootimg_dir, kernel_dir,
@@ -68,8 +77,14 @@ class RootfsPlugin(SourcePlugin):
                                "it is not a valid path, exiting" % part.rootfs_dir)
 
         part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+        pseudo_dir = os.path.join(part.rootfs_dir, "../pseudo")
+        if not os.path.lexists(pseudo_dir):
+            logger.warn("%s folder does not exist. "
+                        "Usernames and permissions will be invalid " % pseudo_dir)
+            pseudo_dir = None
 
         new_rootfs = None
+        new_pseudo = None
         # Handle excluded paths.
         if part.exclude_path or part.include_path:
             # We need a new rootfs directory we can delete files from. Copy to
@@ -78,9 +93,24 @@ class RootfsPlugin(SourcePlugin):
 
             if os.path.lexists(new_rootfs):
                 shutil.rmtree(os.path.join(new_rootfs))
-
             copyhardlinktree(part.rootfs_dir, new_rootfs)
 
+            # Convert the pseudo directory to its new location
+            if (pseudo_dir):
+                new_pseudo = os.path.realpath(
+                             os.path.join(cr_workdir, "pseudo%d" % part.lineno))
+                if os.path.lexists(new_pseudo):
+                    shutil.rmtree(new_pseudo)
+                os.mkdir(new_pseudo)
+                shutil.copy(os.path.join(pseudo_dir, "files.db"),
+                            os.path.join(new_pseudo, "files.db"))
+
+                pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,
+                                                                     new_rootfs,
+                                                                     new_pseudo),
+                                                    part.rootfs_dir, new_rootfs)
+                exec_native_cmd(pseudo_cmd, native_sysroot)
+
             for path in part.include_path or []:
                 copyhardlinktree(path, new_rootfs)
 
@@ -112,4 +142,5 @@ class RootfsPlugin(SourcePlugin):
                     shutil.rmtree(full_path)
 
         part.prepare_rootfs(cr_workdir, oe_builddir,
-                            new_rootfs or part.rootfs_dir, native_sysroot)
+                            new_rootfs or part.rootfs_dir, native_sysroot,
+                            pseudo_dir = new_pseudo or pseudo_dir)
-- 
2.25.1

[PATCH v7 02/10] wic: Fix multi images .wks with bitbake

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

In order to support .wks files with multiple images inside bitbake we
need to explicitly set the pseudo database in use.

Eg: If we try this .mks:
part / --source rootfs --ondisk sda --fstype=ext4
part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4

The username for all the files under /export will be set to the runner
of bitbake (usually UID 1000).

Before we run wic, we need to make sure that the pseudo database will be
flushed, and contains all the data needed.

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 meta/classes/image_types_wic.bbclass | 8 ++++++--
 scripts/lib/wic/partition.py         | 9 +++------
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/meta/classes/image_types_wic.bbclass b/meta/classes/image_types_wic.bbclass
index b83308b45c..96ed0473ee 100644
--- a/meta/classes/image_types_wic.bbclass
+++ b/meta/classes/image_types_wic.bbclass
@@ -32,8 +32,7 @@ IMAGE_CMD_wic () {
 	if [ -z "$wks" ]; then
 		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
 	fi
-
-	BUILDDIR="${TOPDIR}" wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
+	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" ${WIC_CREATE_EXTRA_ARGS}
 	mv "$build_wic/$(basename "${wks%.wks}")"*.direct "$out${IMAGE_NAME_SUFFIX}.wic"
 }
 IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES TOPDIR"
@@ -86,6 +85,10 @@ python do_write_wks_template () {
     bb.utils.copyfile(wks_file, "%s/%s" % (depdir, basename + '-' + os.path.basename(wks_file)))
 }
 
+do_flush_pseudodb() {
+	${FAKEROOTENV} ${FAKEROOTCMD} -S
+}
+
 python () {
     if d.getVar('USING_WIC'):
         wks_file_u = d.getVar('WKS_FULL_PATH', False)
@@ -139,6 +142,7 @@ python do_rootfs_wicenv () {
     depdir = d.getVar('IMGDEPLOYDIR')
     bb.utils.copyfile(os.path.join(outdir, basename) + '.env', os.path.join(depdir, basename) + '.env')
 }
+addtask do_flush_pseudodb after do_image before do_image_wic
 addtask do_rootfs_wicenv after do_image before do_image_wic
 do_rootfs_wicenv[vardeps] += "${WICVARS}"
 do_rootfs_wicenv[prefuncs] = 'set_image_size'
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b02711be37..d850fbd1b1 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -200,13 +200,10 @@ class Partition():
         p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
         if (pseudo_dir == None):
             pseudo_dir = "%s/../pseudo" %  rootfs_dir
-        p_localstatedir = os.environ.get("PSEUDO_LOCALSTATEDIR", pseudo_dir)
-        p_passwd = os.environ.get("PSEUDO_PASSWD", rootfs_dir)
-        p_nosymlinkexp = os.environ.get("PSEUDO_NOSYMLINKEXP", "1")
         pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % p_localstatedir
-        pseudo += "export PSEUDO_PASSWD=%s;" % p_passwd
-        pseudo += "export PSEUDO_NOSYMLINKEXP=%s;" % p_nosymlinkexp
+        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+        pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
         pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
 
         rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
-- 
2.25.1

[PATCH v7 03/10] wic: Add --change-directory argument

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

This option allows to specify which part of a rootfs is going to be
included, the same way the -C argument on tar.

Thanks to this option we can make sure the permissions and usernames
on the target partition are respected, and also simplify the creation of
splitted partitons, not neeting to invoke external vars or using .wks.in
files. Eg:

part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
part /etc --source rootfs --fstype=ext4 --change-directory=etc

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/help.py                  |  6 ++++++
 scripts/lib/wic/ksparser.py              |  1 +
 scripts/lib/wic/partition.py             |  1 +
 scripts/lib/wic/plugins/source/rootfs.py | 21 ++++++++++++++++++---
 4 files changed, 26 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 1e3d06a87b..62a2a90e79 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -980,6 +980,12 @@ DESCRIPTION
                          copies. This option only has an effect with the rootfs
                          source plugin.
 
+         --change-directory: This option is specific to wic. It changes to the
+                             given directory before copying the files. This
+                             option is useful when we want to split a rootfs in
+                             multiple partitions and we want to keep the right
+                             permissions and usernames in all the partitions.
+
          --extra-space: This option is specific to wic. It adds extra
                         space after the space filled by the content
                         of the partition. The final size can go
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 650b976223..c60869d397 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -138,6 +138,7 @@ class KickStart():
         part.add_argument('--align', type=int)
         part.add_argument('--exclude-path', nargs='+')
         part.add_argument('--include-path', nargs='+')
+        part.add_argument('--change-directory')
         part.add_argument("--extra-space", type=sizetype)
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fstype', default='vfat',
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index d850fbd1b1..3240be072a 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -31,6 +31,7 @@ class Partition():
         self.extra_space = args.extra_space
         self.exclude_path = args.exclude_path
         self.include_path = args.include_path
+        self.change_directory = args.change_directory
         self.fsopts = args.fsopts
         self.fstype = args.fstype
         self.label = args.label
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 8b2a067385..85c634f8a1 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -86,14 +86,29 @@ class RootfsPlugin(SourcePlugin):
         new_rootfs = None
         new_pseudo = None
         # Handle excluded paths.
-        if part.exclude_path or part.include_path:
+        if part.exclude_path or part.include_path or part.change_directory:
             # We need a new rootfs directory we can delete files from. Copy to
             # workdir.
             new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
 
             if os.path.lexists(new_rootfs):
                 shutil.rmtree(os.path.join(new_rootfs))
-            copyhardlinktree(part.rootfs_dir, new_rootfs)
+
+            if part.change_directory:
+                cd = part.change_directory
+                if cd[-1] == '/':
+                    cd = cd[:-1]
+                if os.path.isabs(cd):
+                    logger.error("Must be relative: --change-directory=%s" % cd)
+                    sys.exit(1)
+                orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
+                if not orig_dir.startswith(part.rootfs_dir):
+                    logger.error("'%s' points to a path outside the rootfs" % orig_dir)
+                    sys.exit(1)
+
+            else:
+                orig_dir = part.rootfs_dir
+            copyhardlinktree(orig_dir, new_rootfs)
 
             # Convert the pseudo directory to its new location
             if (pseudo_dir):
@@ -108,7 +123,7 @@ class RootfsPlugin(SourcePlugin):
                 pseudo_cmd = "%s -B -m %s -M %s" % (cls.__get_pseudo(native_sysroot,
                                                                      new_rootfs,
                                                                      new_pseudo),
-                                                    part.rootfs_dir, new_rootfs)
+                                                    orig_dir, new_rootfs)
                 exec_native_cmd(pseudo_cmd, native_sysroot)
 
             for path in part.include_path or []:
-- 
2.25.1

[PATCH v7 04/10] wic: Continue if excluded_path does not exist

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

If an excuded path does not exist, continue without an error.
This allows to seamleasly reuse .wks among different projects.

Eg:

part / --source rootfs --fstype=ext4 --exclude-path=opt/private_keys

Where /opt/private_keys in only populated by some of the image.bb files.

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/plugins/source/rootfs.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 85c634f8a1..247f61ff7c 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -136,7 +136,6 @@ class RootfsPlugin(SourcePlugin):
                     sys.exit(1)
 
                 full_path = os.path.realpath(os.path.join(new_rootfs, path))
-
                 # Disallow climbing outside of parent directory using '..',
                 # because doing so could be quite disastrous (we will delete the
                 # directory).
@@ -144,6 +143,9 @@ class RootfsPlugin(SourcePlugin):
                     logger.error("'%s' points to a path outside the rootfs" % orig_path)
                     sys.exit(1)
 
+                if not os.path.lexists(full_path):
+                    continue
+
                 if path.endswith(os.sep):
                     # Delete content only.
                     for entry in os.listdir(full_path):
-- 
2.25.1

[PATCH v7 05/10] wic: Avoid creating invalid pseudo directory

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

If the source of the rootfs is not a bitbake cooked image, or it is not
pointing to the root of one, we call pseudo again, which will produce
a new pseudo folder at rootfs/../pseudo

Eg:

part /etc --source rootfs --rootfs-dir=${IMAGE_ROOTFS}/home --fstype=ext4

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/partition.py | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index 3240be072a..7d9dd616a6 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -199,13 +199,14 @@ class Partition():
         Currently handles ext2/3/4, btrfs, vfat and squashfs.
         """
         p_prefix = os.environ.get("PSEUDO_PREFIX", "%s/usr" % native_sysroot)
-        if (pseudo_dir == None):
-            pseudo_dir = "%s/../pseudo" %  rootfs_dir
-        pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
-        pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
-        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
-        pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
-        pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+        if (pseudo_dir):
+            pseudo = "export PSEUDO_PREFIX=%s;" % p_prefix
+            pseudo += "export PSEUDO_LOCALSTATEDIR=%s;" % pseudo_dir
+            pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+            pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+            pseudo += "%s " % get_bitbake_var("FAKEROOTCMD")
+        else:
+            pseudo = None
 
         rootfs = "%s/rootfs_%s.%s.%s" % (cr_workdir, self.label,
                                          self.lineno, self.fstype)
-- 
2.25.1

[PATCH v7 06/10] oeqa: wic: Add tests for permissions and change-directory

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

Make sure that the permissions and username are respected when using all
the rootfs modifiers.

Add tests for change-directory command

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 meta/lib/oeqa/selftest/cases/wic.py | 90 +++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 626a217e69..41cf23f778 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -62,6 +62,12 @@ def extract_files(debugfs_output):
     return [line.split('/')[5].strip() for line in \
             debugfs_output.strip().split('/\n')]
 
+def files_own_by_root(debugfs_output):
+    for line in debugfs_output.strip().split('/\n'):
+        if line.split('/')[3:5] != ['0', '0']:
+            print(debugfs_output)
+            return False
+    return True
 
 class WicTestCase(OESelftestTestCase):
     """Wic test class."""
@@ -84,6 +90,7 @@ class WicTestCase(OESelftestTestCase):
                 self.skipTest('wic-tools cannot be built due its (intltool|gettext)-native dependency and NLS disable')
 
             bitbake('core-image-minimal')
+            bitbake('core-image-minimal-mtdutils')
             WicTestCase.image_is_ready = True
 
         rmtree(self.resultdir, ignore_errors=True)
@@ -506,6 +513,89 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
                                       % (wks_file, self.resultdir), ignore_status=True).status)
         os.remove(wks_file)
 
+    def test_permissions(self):
+        """Test permissions are respected"""
+
+        oldpath = os.environ['PATH']
+        os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+        t_normal = """
+part / --source rootfs --fstype=ext4
+"""
+        t_exclude = """
+part / --source rootfs --fstype=ext4 --exclude-path=home
+"""
+        t_multi = """
+part / --source rootfs --ondisk sda --fstype=ext4
+part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
+"""
+        t_change = """
+part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/   
+part /etc --source rootfs --fstype=ext4 --change-directory=etc
+"""
+        tests = [t_normal, t_exclude, t_multi, t_change]
+
+        try:
+            for test in tests:
+                include_path = os.path.join(self.resultdir, 'test-include')
+                os.makedirs(include_path)
+                wks_file = os.path.join(include_path, 'temp.wks')
+                with open(wks_file, 'w') as wks:
+                    wks.write(test)
+                runCmd("wic create %s -e core-image-minimal -o %s" \
+                                       % (wks_file, self.resultdir))
+
+                for part in glob(os.path.join(self.resultdir, 'temp-*.direct.p*')):
+                    res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part))
+                    self.assertEqual(True, files_own_by_root(res.output))
+
+                rmtree(self.resultdir, ignore_errors=True)
+
+        finally:
+            os.environ['PATH'] = oldpath
+
+    def test_change_directory(self):
+        """Test --change-directory wks option."""
+
+        oldpath = os.environ['PATH']
+        os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+        try:
+            include_path = os.path.join(self.resultdir, 'test-include')
+            os.makedirs(include_path)
+            wks_file = os.path.join(include_path, 'temp.wks')
+            with open(wks_file, 'w') as wks:
+                wks.write("part /etc --source rootfs --fstype=ext4 --change-directory=etc")
+            runCmd("wic create %s -e core-image-minimal -o %s" \
+                                       % (wks_file, self.resultdir))
+
+            part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+            res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+            files = extract_files(res.output)
+            self.assertIn('passwd', files)
+
+        finally:
+            os.environ['PATH'] = oldpath
+
+    def test_change_directory_errors(self):
+        """Test --change-directory wks option error handling."""
+        wks_file = 'temp.wks'
+
+        # Absolute argument.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --fstype=ext4 --change-directory /usr")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                      % (wks_file, self.resultdir), ignore_status=True).status)
+        os.remove(wks_file)
+
+        # Argument pointing to parent directory.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --fstype=ext4 --change-directory ././..")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                      % (wks_file, self.resultdir), ignore_status=True).status)
+        os.remove(wks_file)
+
 class Wic2(WicTestCase):
 
     def test_bmap_short(self):
-- 
2.25.1

[PATCH v7 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

Executables like tar won't be available on the native sysroot, as they
are part of the ALREADY_PROVIDED variable.

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/misc.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/wic/misc.py b/scripts/lib/wic/misc.py
index 1f199b9f23..91975ba151 100644
--- a/scripts/lib/wic/misc.py
+++ b/scripts/lib/wic/misc.py
@@ -45,7 +45,8 @@ NATIVE_RECIPES = {"bmaptool": "bmap-tools",
                   "parted": "parted",
                   "sfdisk": "util-linux",
                   "sgdisk": "gptfdisk",
-                  "syslinux": "syslinux"
+                  "syslinux": "syslinux",
+                  "tar": "tar"
                  }
 
 def runtool(cmdln_or_args):
@@ -112,6 +113,15 @@ def exec_cmd(cmd_and_args, as_shell=False):
     """
     return _exec_cmd(cmd_and_args, as_shell)[1]
 
+def find_executable(cmd, paths):
+    recipe = cmd
+    if recipe in NATIVE_RECIPES:
+        recipe =  NATIVE_RECIPES[recipe]
+    provided = get_bitbake_var("ASSUME_PROVIDED")
+    if provided and "%s-native" % recipe in provided:
+        return True
+
+    return spawn.find_executable(cmd, paths)
 
 def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
     """
@@ -136,7 +146,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, pseudo=""):
     logger.debug("exec_native_cmd: %s", native_cmd_and_args)
 
     # If the command isn't in the native sysroot say we failed.
-    if spawn.find_executable(args[0], native_paths):
+    if find_executable(args[0], native_paths):
         ret, out = _exec_cmd(native_cmd_and_args, True)
     else:
         ret = 127
-- 
2.25.1

Re: [OE-core] [PATCH v7 07/10] wic: misc: Do not find for executables in ALREADY_PROVIDED

[Richard Purdie]

On Sun, 2020-04-19 at 08:35 +0200, Ricardo Ribalda wrote:
> From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>
> 
> Executables like tar won't be available on the native sysroot, as
> they
> are part of the ALREADY_PROVIDED variable.

ASSUME_PROVIDED :)

I fixed the commit message in -next.

Cheers,

Richard

[PATCH v7 08/10] wic: root: Add an opt. destination on include-path

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

Allow specifying an optional destination to include-path and make the
option aware of permissions and owners.

It is very useful for making a partition that contains the rootfs for a
host and a target Eg:

/ -> Roofs for the host
/export/ -> Rootfs for the target (which will netboot)

Although today we support making a partition for "/export" this might
not be compatible with some upgrade systems, or we might be limited by
the number of partitions.

With this patch we can use something like:

part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ --include-path hello

on the .wks file.

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 meta/recipes-core/meta/wic-tools.bb      |  2 +-
 scripts/lib/wic/help.py                  | 20 ++++----
 scripts/lib/wic/ksparser.py              |  2 +-
 scripts/lib/wic/plugins/source/rootfs.py | 60 +++++++++++++++++++++++-
 4 files changed, 72 insertions(+), 12 deletions(-)

diff --git a/meta/recipes-core/meta/wic-tools.bb b/meta/recipes-core/meta/wic-tools.bb
index 09eb409e87..8aeb942ed2 100644
--- a/meta/recipes-core/meta/wic-tools.bb
+++ b/meta/recipes-core/meta/wic-tools.bb
@@ -6,7 +6,7 @@ DEPENDS = "\
            parted-native syslinux-native gptfdisk-native dosfstools-native \
            mtools-native bmap-tools-native grub-efi-native cdrtools-native \
            btrfs-tools-native squashfs-tools-native pseudo-native \
-           e2fsprogs-native util-linux-native \
+           e2fsprogs-native util-linux-native tar-native\
            "
 DEPENDS_append_x86 = " syslinux grub-efi systemd-boot"
 DEPENDS_append_x86-64 = " syslinux grub-efi systemd-boot"
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 62a2a90e79..bd3a2b97df 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -971,14 +971,18 @@ DESCRIPTION
                          has an effect with the rootfs source plugin.
 
          --include-path: This option is specific to wic. It adds the contents
-                         of the given path to the resulting image. The path is
-                         relative to the directory in which wic is running not
-                         the rootfs itself so use of an absolute path is
-                         recommended. This option is most useful when multiple
-                         copies of the rootfs are added to an image and it is
-                         required to add extra content to only one of these
-                         copies. This option only has an effect with the rootfs
-                         source plugin.
+                         of the given path or a rootfs to the resulting image.
+                         The option contains two fields, the origin and the
+                         destination. When the origin is a rootfs, it follows
+                         the same logic as the rootfs-dir argument and the
+                         permissions and owners are kept. When the origin is a
+                         path, it is relative to the directory in which wic is
+                         running not the rootfs itself so use of an absolute
+                         path is recommended, and the owner and group is set to
+                         root:root. If no destination is given it is
+                         automatically set to the root of the rootfs. This
+                         option only has an effect with the rootfs source
+                         plugin.
 
          --change-directory: This option is specific to wic. It changes to the
                              given directory before copying the files. This
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index c60869d397..b8befe78e3 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -137,7 +137,7 @@ class KickStart():
         part.add_argument('--active', action='store_true')
         part.add_argument('--align', type=int)
         part.add_argument('--exclude-path', nargs='+')
-        part.add_argument('--include-path', nargs='+')
+        part.add_argument('--include-path', nargs='+', action='append')
         part.add_argument('--change-directory')
         part.add_argument("--extra-space", type=sizetype)
         part.add_argument('--fsoptions', dest='fsopts')
diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 247f61ff7c..544e868b5e 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -17,6 +17,7 @@ import shutil
 import sys
 
 from oe.path import copyhardlinktree
+from pathlib import Path
 
 from wic import WicError
 from wic.pluginbase import SourcePlugin
@@ -126,8 +127,63 @@ class RootfsPlugin(SourcePlugin):
                                                     orig_dir, new_rootfs)
                 exec_native_cmd(pseudo_cmd, native_sysroot)
 
-            for path in part.include_path or []:
-                copyhardlinktree(path, new_rootfs)
+            for in_path in part.include_path or []:
+                #parse arguments
+                include_path = in_path[0]
+                if len(in_path) > 2:
+                    logger.error("'Invalid number of arguments for include-path")
+                    sys.exit(1)
+                if len(in_path) == 2:
+                    path = in_path[1]
+                else:
+                    path = None
+
+                # Pack files to be included into a tar file.
+                # We need to create a tar file, because that way we can keep the
+                # permissions from the files even when they belong to different
+                # pseudo enviroments.
+                # If we simply copy files using copyhardlinktree/copytree... the
+                # copied files will belong to the user running wic.
+                tar_file = os.path.realpath(
+                           os.path.join(cr_workdir, "include-path%d.tar" % part.lineno))
+                if os.path.isfile(include_path):
+                    parent = os.path.dirname(os.path.realpath(include_path))
+                    tar_cmd = "tar c --owner=root --group=root -f %s -C %s %s" % (
+                                tar_file, parent, os.path.relpath(include_path, parent))
+                    exec_native_cmd(tar_cmd, native_sysroot)
+                else:
+                    if include_path in krootfs_dir:
+                        include_path = krootfs_dir[include_path]
+                    include_path = cls.__get_rootfs_dir(include_path)
+                    include_pseudo = os.path.join(include_path, "../pseudo")
+                    if os.path.lexists(include_pseudo):
+                        pseudo = cls.__get_pseudo(native_sysroot, include_path,
+                                                  include_pseudo)
+                        tar_cmd = "tar cf %s -C %s ." % (tar_file, include_path)
+                    else:
+                        pseudo = None
+                        tar_cmd = "tar c --owner=root --group=root -f %s -C %s ." % (
+                                tar_file, include_path)
+                    exec_native_cmd(tar_cmd, native_sysroot, pseudo)
+
+                #create destination
+                if path:
+                    destination = os.path.realpath(os.path.join(new_rootfs, path))
+                    if not destination.startswith(new_rootfs):
+                        logger.error("%s %s" % (destination, new_rootfs))
+                        sys.exit(1)
+                    Path(destination).mkdir(parents=True, exist_ok=True)
+                else:
+                    destination = new_rootfs
+
+                #extract destination
+                untar_cmd = "tar xf %s -C %s" % (tar_file, destination)
+                if new_pseudo:
+                    pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)
+                else:
+                    pseudo = None
+                exec_native_cmd(untar_cmd, native_sysroot, pseudo)
+                os.remove(tar_file)
 
             for orig_path in part.exclude_path or []:
                 path = orig_path
-- 
2.25.1

[PATCH v7 09/10] wic: rootfs: Combine path_validation in one function

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

Combine all the common path validation in a function to avoid code
duplication.

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 scripts/lib/wic/plugins/source/rootfs.py | 41 +++++++++++-------------
 1 file changed, 19 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py
index 544e868b5e..f1db83f8a1 100644
--- a/scripts/lib/wic/plugins/source/rootfs.py
+++ b/scripts/lib/wic/plugins/source/rootfs.py
@@ -32,6 +32,22 @@ class RootfsPlugin(SourcePlugin):
 
     name = 'rootfs'
 
+    @staticmethod
+    def __validate_path(cmd, rootfs_dir, path):
+        if os.path.isabs(path):
+            logger.error("%s: Must be relative: %s" % (cmd, orig_path))
+            sys.exit(1)
+
+        # Disallow climbing outside of parent directory using '..',
+        # because doing so could be quite disastrous (we will delete the
+        # directory, or modify a directory outside OpenEmbedded).
+        full_path = os.path.realpath(os.path.join(rootfs_dir, path))
+        if not full_path.startswith(os.path.realpath(rootfs_dir)):
+            logger.error("%s: Must point inside the rootfs:" % (cmd, path))
+            sys.exit(1)
+
+        return full_path
+
     @staticmethod
     def __get_rootfs_dir(rootfs_dir):
         if os.path.isdir(rootfs_dir):
@@ -99,14 +115,7 @@ class RootfsPlugin(SourcePlugin):
                 cd = part.change_directory
                 if cd[-1] == '/':
                     cd = cd[:-1]
-                if os.path.isabs(cd):
-                    logger.error("Must be relative: --change-directory=%s" % cd)
-                    sys.exit(1)
-                orig_dir = os.path.realpath(os.path.join(part.rootfs_dir, cd))
-                if not orig_dir.startswith(part.rootfs_dir):
-                    logger.error("'%s' points to a path outside the rootfs" % orig_dir)
-                    sys.exit(1)
-
+                orig_dir = cls.__validate_path("--change-directory", part.rootfs_dir, cd)
             else:
                 orig_dir = part.rootfs_dir
             copyhardlinktree(orig_dir, new_rootfs)
@@ -168,10 +177,7 @@ class RootfsPlugin(SourcePlugin):
 
                 #create destination
                 if path:
-                    destination = os.path.realpath(os.path.join(new_rootfs, path))
-                    if not destination.startswith(new_rootfs):
-                        logger.error("%s %s" % (destination, new_rootfs))
-                        sys.exit(1)
+                    destination = cls.__validate_path("--include-path", new_rootfs, path)
                     Path(destination).mkdir(parents=True, exist_ok=True)
                 else:
                     destination = new_rootfs
@@ -187,17 +193,8 @@ class RootfsPlugin(SourcePlugin):
 
             for orig_path in part.exclude_path or []:
                 path = orig_path
-                if os.path.isabs(path):
-                    logger.error("Must be relative: --exclude-path=%s" % orig_path)
-                    sys.exit(1)
 
-                full_path = os.path.realpath(os.path.join(new_rootfs, path))
-                # Disallow climbing outside of parent directory using '..',
-                # because doing so could be quite disastrous (we will delete the
-                # directory).
-                if not full_path.startswith(new_rootfs):
-                    logger.error("'%s' points to a path outside the rootfs" % orig_path)
-                    sys.exit(1)
+                full_path = cls.__validate_path("--exclude-path", new_rootfs, path)
 
                 if not os.path.lexists(full_path):
                     continue
-- 
2.25.1

[PATCH v7 10/10] oeqa: wic: Add more tests for include_path

[Ricardo Ribalda]

From: Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>

Make sure permissions are respected.

Add new test for orig/destination option.

Cc: Paul Barker <pbarker@konsulko.com>
Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com>
---
 meta/lib/oeqa/selftest/cases/wic.py | 63 ++++++++++++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 41cf23f778..c8765e5330 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -486,15 +486,76 @@ part /part2 --source rootfs --ondisk mmcblk0 --fstype=ext4 --include-path %s"""
             res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
             files = extract_files(res.output)
             self.assertNotIn('test-file', files)
+            self.assertEqual(True, files_own_by_root(res.output))
 
-            # Test partition 2, should not contain 'test-file'
+            # Test partition 2, should contain 'test-file'
             res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part2))
             files = extract_files(res.output)
             self.assertIn('test-file', files)
+            self.assertEqual(True, files_own_by_root(res.output))
 
         finally:
             os.environ['PATH'] = oldpath
 
+    def test_include_path_embeded(self):
+        """Test --include-path wks option."""
+
+        oldpath = os.environ['PATH']
+        os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
+
+        try:
+            include_path = os.path.join(self.resultdir, 'test-include')
+            os.makedirs(include_path)
+            with open(os.path.join(include_path, 'test-file'), 'w') as t:
+                t.write("test\n")
+            wks_file = os.path.join(include_path, 'temp.wks')
+            with open(wks_file, 'w') as wks:
+                wks.write("""
+part / --source rootfs  --fstype=ext4 --include-path %s --include-path core-image-minimal-mtdutils export/"""
+                          % (include_path))
+            runCmd("wic create %s -e core-image-minimal -o %s" \
+                                       % (wks_file, self.resultdir))
+
+            part1 = glob(os.path.join(self.resultdir, 'temp-*.direct.p1'))[0]
+
+            res = runCmd("debugfs -R 'ls -p' %s 2>/dev/null" % (part1))
+            files = extract_files(res.output)
+            self.assertIn('test-file', files)
+            self.assertEqual(True, files_own_by_root(res.output))
+
+            res = runCmd("debugfs -R 'ls -p /export/etc/' %s 2>/dev/null" % (part1))
+            files = extract_files(res.output)
+            self.assertIn('passwd', files)
+            self.assertEqual(True, files_own_by_root(res.output))
+
+        finally:
+            os.environ['PATH'] = oldpath
+
+    def test_include_path_errors(self):
+        """Test --include-path wks option error handling."""
+        wks_file = 'temp.wks'
+
+        # Absolute argument.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils /export")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                      % (wks_file, self.resultdir), ignore_status=True).status)
+        os.remove(wks_file)
+
+        # Argument pointing to parent directory.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils ././..")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                      % (wks_file, self.resultdir), ignore_status=True).status)
+        os.remove(wks_file)
+
+        # 3 Argument pointing to parent directory.
+        with open(wks_file, 'w') as wks:
+            wks.write("part / --source rootfs --fstype=ext4 --include-path core-image-minimal-mtdutils export/ dummy")
+        self.assertNotEqual(0, runCmd("wic create %s -e core-image-minimal -o %s" \
+                                      % (wks_file, self.resultdir), ignore_status=True).status)
+        os.remove(wks_file)
+
     def test_exclude_path_errors(self):
         """Test --exclude-path wks option error handling."""
         wks_file = 'temp.wks'
-- 
2.25.1

Re: [PATCH v7 00/10] Fix permissions and embed-rotofs

[Paul Barker]

On Sun, 19 Apr 2020 08:35:28 +0200
Ricardo Ribalda Delgado <ricardo@ribalda.com> wrote:

> Today wic behaves differently if we run it from bitbake of directly from the
> commandline.
> 
> When it is run from bitbake, the permissions/usersnames are handled by the
> pseudo database of the main image.
> 
> When it is run from the comandline ,it is run outside the main image database.
> 
> This results on permissions/usernames not working ok on some usecases on both
> bitbake and wic.
> 
> 
> This is an attempt to fix all the permission bugs that I am aware from wic.
> Using the following usecases:
> 
> 
> #exclude-path
> part / --source rootfs --fstype=ext4 --exclude-path=home
> 
> #split_partition
> part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
> part /etc --source rootfs --rootfs-dir=tmp/work/qt5222-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/ --fstype=ext4
> 
> #multi_partition
> part / --source rootfs --ondisk sda --fstype=ext4
> part /export --source rootfs --rootfs=core-image-minimal-mtdutils --fstype=ext4
> 
> 
> With the current master:
> -------------------------
> #exclude-path
> From Bitbake: OK
> wic cmdline: FAIL
> 
> #split_partition
> From Bitbake: OK
> wic cmdline: FAIL, permissions invalid on both partitions
> 
> #multi_partition
> From Bitbake: FAIL second partition
> wic cmdline: OK
> 
> 
> After:  wic: Fix permissions when using exclude or include path
> --------------------------------------------------------------------
> #exclude-path
> From Bitbake: OK
> wic cmdline: OK
> 
> #split_partition
> From Bitbake: OK
> wic cmdline: FAIL, permissions invalid on second partition
> 
> #multi_partition
> From Bitbake: FAIL second partition
> wic cmdline: OK
> 
> 
> After: wic: Fix multi images .wks with bitbake
> ----------------------------------------------
> #exclude-path
> From Bitbake: OK
> wic cmdline: OK
> 
> #split_partition
> From Bitbake: FAIL, permissions invalid on second partition
> wic cmdline: FAIL, permissions invalid on second partition
> 
> #multi_partition
> From Bitbake: OK
> wic cmdline: OK
> 
> 
> 
> After: wic: Add --change-directory argument
> --------------------------------------------
> 
> we can have a .wks like:
> part / --source rootfs --ondisk sda --fstype=ext4 --exclude-path=etc/
> part /etc --source rootfs --fstype=ext4 --change-directory=etc
> 
> 
> Wich works fine from bitbake and from cmdline, and has the same functionality as
> split_partition.
> 
> 
> Changelog v6->v7 (Thanks Paul):
>   - Use passwd instead of shadow for oetests
>   - For pseudo folder inside workdir
>   - Improve comments on tar
>   - Use line number on tar name
>   - Allow include-files of files in workdir
> 
> Changelog v5->v6:
>   - Allow two arguments to include-path
>   - Fix permission on include-path
>   - exec_native with ASSUME_PROVIDED
>   - More unittesting
> 
> Changelog v4->v5:
>   - Add unittesting
>   - wic: Avoid creating invalid pseudo directory
> 
> Ricardo Ribalda Delgado (10):
>   wic: Fix permissions when using exclude or include path
>   wic: Fix multi images .wks with bitbake
>   wic: Add --change-directory argument
>   wic: Continue if excluded_path does not exist
>   wic: Avoid creating invalid pseudo directory
>   oeqa: wic: Add tests for permissions and change-directory
>   wic: misc: Do not find for executables in ALREADY_PROVIDED
>   wic: root: Add an opt. destination on include-path
>   wic: rootfs: Combine path_validation in one function
>   oeqa: wic: Add more tests for include_path
> 
>  meta/classes/image_types_wic.bbclass     |   8 +-
>  meta/lib/oeqa/selftest/cases/wic.py      | 153 ++++++++++++++++++++++-
>  meta/recipes-core/meta/wic-tools.bb      |   2 +-
>  scripts/lib/wic/help.py                  |  26 ++--
>  scripts/lib/wic/ksparser.py              |   3 +-
>  scripts/lib/wic/misc.py                  |  14 ++-
>  scripts/lib/wic/partition.py             |  20 +--
>  scripts/lib/wic/plugins/source/rootfs.py | 133 +++++++++++++++++---
>  8 files changed, 318 insertions(+), 41 deletions(-)
> 

All looks good to me now, thanks for iterating on this :)

My knowledge on pseudo is a little limited so I can't guarantee I've caught
all the implications of these changes on that front. You've included tests
and it's going to go into master at the start of a development cycle so we'll
have time for anything I've missed to get shaken out. I think it's good to
merge now.

-- 
Paul Barker
Konsulko Group

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Richard Purdie]

On Sun, 2020-04-19 at 08:35 +0200, Ricardo Ribalda wrote:
> 
> Changelog v6->v7 (Thanks Paul):
>   - Use passwd instead of shadow for oetests
>   - For pseudo folder inside workdir
>   - Improve comments on tar
>   - Use line number on tar name
>   - Allow include-files of files in workdir
> 
> Changelog v5->v6:
>   - Allow two arguments to include-path
>   - Fix permission on include-path
>   - exec_native with ASSUME_PROVIDED
>   - More unittesting
> 
> Changelog v4->v5:
>   - Add unittesting
>   - wic: Avoid creating invalid pseudo directory
> 
> Ricardo Ribalda Delgado (10):
>   wic: Fix permissions when using exclude or include path
>   wic: Fix multi images .wks with bitbake
>   wic: Add --change-directory argument
>   wic: Continue if excluded_path does not exist
>   wic: Avoid creating invalid pseudo directory
>   oeqa: wic: Add tests for permissions and change-directory
>   wic: misc: Do not find for executables in ALREADY_PROVIDED
>   wic: root: Add an opt. destination on include-path
>   wic: rootfs: Combine path_validation in one function
>   oeqa: wic: Add more tests for include_path

I did run this through the autobuilder along with other changes and
there were 5 oe-selftest failures for wic:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/858
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/855
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/861
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/849

Cheers,

Richard

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Hi Richard

Thanks for the test.

All the errors seem to have the same pattern:

FileNotFoundError: [Errno 2] No such file or directory:
'/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-27135/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-r0/testimage/qemurunner_log.20200423015612'

I do not believe that it is related to my patchset, I have seen that
you are already running the selftest without the patchset
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/850

Lets see if it also crashes or not. I am also trying to replicate locally.

Best regards!

On Thu, Apr 23, 2020 at 10:54 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sun, 2020-04-19 at 08:35 +0200, Ricardo Ribalda wrote:
> >
> > Changelog v6->v7 (Thanks Paul):
> >   - Use passwd instead of shadow for oetests
> >   - For pseudo folder inside workdir
> >   - Improve comments on tar
> >   - Use line number on tar name
> >   - Allow include-files of files in workdir
> >
> > Changelog v5->v6:
> >   - Allow two arguments to include-path
> >   - Fix permission on include-path
> >   - exec_native with ASSUME_PROVIDED
> >   - More unittesting
> >
> > Changelog v4->v5:
> >   - Add unittesting
> >   - wic: Avoid creating invalid pseudo directory
> >
> > Ricardo Ribalda Delgado (10):
> >   wic: Fix permissions when using exclude or include path
> >   wic: Fix multi images .wks with bitbake
> >   wic: Add --change-directory argument
> >   wic: Continue if excluded_path does not exist
> >   wic: Avoid creating invalid pseudo directory
> >   oeqa: wic: Add tests for permissions and change-directory
> >   wic: misc: Do not find for executables in ALREADY_PROVIDED
> >   wic: root: Add an opt. destination on include-path
> >   wic: rootfs: Combine path_validation in one function
> >   oeqa: wic: Add more tests for include_path
>
> I did run this through the autobuilder along with other changes and
> there were 5 oe-selftest failures for wic:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/858
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/855
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/861
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/849
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Richard Purdie]

On Thu, 2020-04-23 at 11:57 +0200, Ricardo Ribalda Delgado wrote:
> Hi Richard
> 
> Thanks for the test.
> 
> All the errors seem to have the same pattern:
> 
> FileNotFoundError: [Errno 2] No such file or directory:
> '/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-
> 27135/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-
> r0/testimage/qemurunner_log.20200423015612'
> 
> I do not believe that it is related to my patchset, I have seen that
> you are already running the selftest without the patchset
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/850
> 
> Lets see if it also crashes or not. I am also trying to replicate
> locally.

It did recur:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/862

so your patches are probably not the cause, sorry! :)

I'll continue to try and track it down.

Cheers,

Richard

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

On Thu, Apr 23, 2020 at 12:45 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2020-04-23 at 11:57 +0200, Ricardo Ribalda Delgado wrote:
> > Hi Richard
> >
> > Thanks for the test.
> >
> > All the errors seem to have the same pattern:
> >
> > FileNotFoundError: [Errno 2] No such file or directory:
> > '/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-
> > 27135/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-
> > r0/testimage/qemurunner_log.20200423015612'
> >
> > I do not believe that it is related to my patchset, I have seen that
> > you are already running the selftest without the patchset
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/850
> >
> > Lets see if it also crashes or not. I am also trying to replicate
> > locally.
>
> It did recur:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/862
>
> so your patches are probably not the cause, sorry! :)

No problem, thanks!

I am trying to replicate it anyway here. If I find a problem I will ping you.

>
> I'll continue to try and track it down.
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

On my setup, using debian crops the error is even different :(

If you want I can keep trying, otherwise I will leave it to the experts ;)


Best regards!

pokyuser@e81a7801004c:/workdir/build$ oe-selftest  -r
wic.Wic2.test_biosplusefi_plugin_qemu wic.Wic2.test_expand_mbr_image
wic.Wic2.test_qemu wic.Wic2.test_qemu_efi
wic.Wic2.test_rawcopy_plugin_qemu
2020-04-23 09:50:45,488 - oe-selftest - WARNING - meta-selftest layer
not found in BBLAYERS, adding it
2020-04-23 09:50:50,587 - oe-selftest - ERROR - Please unset
SANITY_TESTED_DISTROS in order to run oe-selftest
pokyuser@e81a7801004c:/workdir/build$ oe-selftest  -r
wic.Wic2.test_biosplusefi_plugin_qemu wic.Wic2.test_expand_mbr_image
wic.Wic2.test_qemu wic.Wic2.test_qemu_efi
wic.Wic2.test_rawcopy_plugin_qemu
2020-04-23 09:51:46,909 - oe-selftest - INFO - Adding layer libraries:
2020-04-23 09:51:46,910 - oe-selftest - INFO - /workdir/meta/lib
2020-04-23 09:51:46,910 - oe-selftest - INFO - /workdir/meta-yocto-bsp/lib
2020-04-23 09:51:46,910 - oe-selftest - INFO - /workdir/meta-selftest/lib
2020-04-23 09:51:46,912 - oe-selftest - INFO - Running bitbake -e to
test the configuration is valid/parsable
2020-04-23 09:51:50,683 - oe-selftest - INFO - Adding: "include
selftest.inc" in /workdir/build-st/conf/local.conf
2020-04-23 09:51:50,684 - oe-selftest - INFO - Adding: "include
bblayers.inc" in bblayers.conf
2020-04-23 09:51:50,685 - oe-selftest - INFO -
test_biosplusefi_plugin_qemu (wic.Wic2)
2020-04-23 11:15:10,685 - oe-selftest - INFO - Keepalive message
2020-04-23 12:38:30,687 - oe-selftest - INFO - Keepalive message
2020-04-23 12:46:55,356 - oe-selftest - INFO -  ... ERROR

Stdout:
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423124632
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423124632
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:57999
Created listening socket for qemu serial console on: 127.0.0.1:45643
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=57999:45643 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2137514
waiting at most 120 seconds for qemu pid (04/23/20 12:46:35)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231246_qemu
2020-04-23 12:46:55,357 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 732, in
test_biosplusefi_plugin_qemu
    with runqemu('core-image-minimal', ssh=False, image_fstype='wic') as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423124632
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423124632
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:57999
Created listening socket for qemu serial console on: 127.0.0.1:45643
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=57999:45643 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2137514
waiting at most 120 seconds for qemu pid (04/23/20 12:46:35)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231246_qemu

2020-04-23 12:46:55,357 - oe-selftest - INFO - test_expand_mbr_image (wic.Wic2)
2020-04-23 12:48:53,064 - oe-selftest - INFO -  ... ERROR

Stdout:
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.ext4
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423124830
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423124830
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:39485
Created listening socket for qemu serial console on: 127.0.0.1:51903
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.ext4
tcpserial=39485:51903 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2144564
waiting at most 120 seconds for qemu pid (04/23/20 12:48:32)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231248_qemu
2020-04-23 12:48:53,065 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 1002, in
test_expand_mbr_image
    with runqemu('core-image-minimal', ssh=False) as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.ext4
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423124830
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423124830
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:39485
Created listening socket for qemu serial console on: 127.0.0.1:51903
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.ext4
tcpserial=39485:51903 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2144564
waiting at most 120 seconds for qemu pid (04/23/20 12:48:32)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231248_qemu

2020-04-23 12:48:53,065 - oe-selftest - INFO - test_qemu (wic.Wic2)
2020-04-23 12:50:55,372 - oe-selftest - INFO -  ... ERROR

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423125032
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/ssh_target_log.20200423125032
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:41315
Created listening socket for qemu serial console on: 127.0.0.1:58191
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
tcpserial=41315:58191 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2152208
waiting at most 120 seconds for qemu pid (04/23/20 12:50:35)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231250_qemu
2020-04-23 12:50:55,373 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 605, in test_qemu
    with runqemu('wic-image-minimal', ssh=False) as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423125032
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/ssh_target_log.20200423125032
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:41315
Created listening socket for qemu serial console on: 127.0.0.1:58191
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
tcpserial=41315:58191 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2152208
waiting at most 120 seconds for qemu pid (04/23/20 12:50:35)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231250_qemu

2020-04-23 12:50:55,373 - oe-selftest - INFO - test_qemu_efi (wic.Wic2)
2020-04-23 13:03:49,786 - oe-selftest - INFO -  ... ERROR

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423130327
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423130327
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:41257
Created listening socket for qemu serial console on: 127.0.0.1:53105
launchcmd=runqemu snapshot nographic ovmf qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=41257:53105 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2168394
waiting at most 120 seconds for qemu pid (04/23/20 13:03:29)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231303_qemu
2020-04-23 13:03:49,787 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 625, in
test_qemu_efi
    runqemuparams='ovmf', image_fstype='wic') as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423130327
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423130327
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:41257
Created listening socket for qemu serial console on: 127.0.0.1:53105
launchcmd=runqemu snapshot nographic ovmf qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=41257:53105 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2168394
waiting at most 120 seconds for qemu pid (04/23/20 13:03:29)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231303_qemu

2020-04-23 13:03:49,787 - oe-selftest - INFO -
test_rawcopy_plugin_qemu (wic.Wic2)
2020-04-23 13:07:15,387 - oe-selftest - INFO -  ... ERROR

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423130652
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423130652
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:39627
Created listening socket for qemu serial console on: 127.0.0.1:37345
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=39627:37345 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2182265
waiting at most 120 seconds for qemu pid (04/23/20 13:06:55)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231306_qemu
2020-04-23 13:07:15,387 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 702, in
test_rawcopy_plugin_qemu
    with runqemu('core-image-minimal', ssh=False, image_fstype='wic') as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423130652
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423130652
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:39627
Created listening socket for qemu serial console on: 127.0.0.1:37345
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=39627:37345 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2182265
waiting at most 120 seconds for qemu pid (04/23/20 13:06:55)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231306_qemu

2020-04-23 13:07:15,388 - oe-selftest - INFO -
======================================================================
2020-04-23 13:07:15,389 - oe-selftest - INFO - ERROR:
test_biosplusefi_plugin_qemu (wic.Wic2)
2020-04-23 13:07:15,389 - oe-selftest - INFO -
----------------------------------------------------------------------
2020-04-23 13:07:15,389 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 732, in
test_biosplusefi_plugin_qemu
    with runqemu('core-image-minimal', ssh=False, image_fstype='wic') as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423124632
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423124632
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:57999
Created listening socket for qemu serial console on: 127.0.0.1:45643
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=57999:45643 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2137514
waiting at most 120 seconds for qemu pid (04/23/20 12:46:35)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231246_qemu
======================================================================
2020-04-23 13:07:15,389 - oe-selftest - INFO - ERROR:
test_expand_mbr_image (wic.Wic2)
2020-04-23 13:07:15,390 - oe-selftest - INFO -
----------------------------------------------------------------------
2020-04-23 13:07:15,390 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 1002, in
test_expand_mbr_image
    with runqemu('core-image-minimal', ssh=False) as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.ext4
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423124830
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423124830
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:39485
Created listening socket for qemu serial console on: 127.0.0.1:51903
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.ext4
tcpserial=39485:51903 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2144564
waiting at most 120 seconds for qemu pid (04/23/20 12:48:32)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231248_qemu
======================================================================
2020-04-23 13:07:15,390 - oe-selftest - INFO - ERROR: test_qemu (wic.Wic2)
2020-04-23 13:07:15,391 - oe-selftest - INFO -
----------------------------------------------------------------------
2020-04-23 13:07:15,391 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 605, in test_qemu
    with runqemu('wic-image-minimal', ssh=False) as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423125032
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/testimage/ssh_target_log.20200423125032
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:41315
Created listening socket for qemu serial console on: 127.0.0.1:58191
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/wic-image-minimal-qemux86-64.wic
tcpserial=41315:58191 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2152208
waiting at most 120 seconds for qemu pid (04/23/20 12:50:35)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231250_qemu
======================================================================
2020-04-23 13:07:15,391 - oe-selftest - INFO - ERROR: test_qemu_efi (wic.Wic2)
2020-04-23 13:07:15,392 - oe-selftest - INFO -
----------------------------------------------------------------------
2020-04-23 13:07:15,392 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 625, in
test_qemu_efi
    runqemuparams='ovmf', image_fstype='wic') as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423130327
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423130327
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:41257
Created listening socket for qemu serial console on: 127.0.0.1:53105
launchcmd=runqemu snapshot nographic ovmf qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=41257:53105 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2168394
waiting at most 120 seconds for qemu pid (04/23/20 13:03:29)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231303_qemu
======================================================================
2020-04-23 13:07:15,392 - oe-selftest - INFO - ERROR:
test_rawcopy_plugin_qemu (wic.Wic2)
2020-04-23 13:07:15,392 - oe-selftest - INFO -
----------------------------------------------------------------------
2020-04-23 13:07:15,393 - oe-selftest - INFO - Traceback (most recent
call last):
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 42, in wrapped_f
    return func(*args, **kwargs)
  File "/workdir/meta/lib/oeqa/selftest/cases/wic.py", line 702, in
test_rawcopy_plugin_qemu
    with runqemu('core-image-minimal', ssh=False, image_fstype='wic') as qemu:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/workdir/meta/lib/oeqa/utils/commands.py", line 342, in runqemu
    qemu.start(params=qemuparams, ssh=ssh,
runqemuparams=runqemuparams, launch_cmd=launch_cmd,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/targetcontrol.py", line 167, in start
    start = self.runner.start(params, get_ip=ssh,
extra_bootparams=extra_bootparams, runqemuparams=runqemuparams,
discard_writes=discard_writes)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 166, in start
    return self.launch(launch_cmd, qemuparams=qemuparams,
get_ip=get_ip, extra_bootparams=extra_bootparams, env=env)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 245, in launch
    time.sleep(0.5)
  File "/workdir/meta/lib/oeqa/utils/qemurunner.py", line 128, in handleSIGCHLD
    raise SystemExit
SystemExit

Stdout:
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection (#1)...
Loading cache...done.
rootfs file: /workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
Qemu log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20200423130652
SSH log file: /workdir/build-st/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20200423130652
Not using kvm for runqemu
Created listening socket for qemu serial console on: 127.0.0.1:39627
Created listening socket for qemu serial console on: 127.0.0.1:37345
launchcmd=runqemu snapshot nographic  qemux86-64
/workdir/build-st/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic
tcpserial=39627:37345 bootparams="console=tty1 console=ttyS0,115200n8
printk.time=1" qemuparams="-pidfile pidfile_135"
runqemu started, pid is 2182265
waiting at most 120 seconds for qemu pid (04/23/20 13:06:55)
runqemu exited with code 1
Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

Sending SIGTERM to runqemu
Qemu ended unexpectedly, dump data from host is in
/workdir/build-st/tmp/log/runtime-hostdump/202004231306_qemu
----------------------------------------------------------------------
2020-04-23 13:07:15,393 - oe-selftest - INFO - Ran 5 tests in 11724.705s
2020-04-23 13:07:15,393 - oe-selftest - INFO - FAILED
2020-04-23 13:07:15,394 - oe-selftest - INFO -  (errors=5)
2020-04-23 13:07:20,028 - oe-selftest - INFO - RESULTS:
2020-04-23 13:07:20,030 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_biosplusefi_plugin_qemu: ERROR (10504.67s)
2020-04-23 13:07:20,030 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_expand_mbr_image: ERROR (117.71s)
2020-04-23 13:07:20,030 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_qemu: ERROR (122.31s)
2020-04-23 13:07:20,031 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_qemu_efi: ERROR (774.41s)
2020-04-23 13:07:20,031 - oe-selftest - INFO - RESULTS -
wic.Wic2.test_rawcopy_plugin_qemu: ERROR (205.60s)
2020-04-23 13:07:20,033 - oe-selftest - INFO - SUMMARY:
2020-04-23 13:07:20,033 - oe-selftest - INFO - oe-selftest () - Ran 5
tests in 11726.658s
2020-04-23 13:07:20,033 - oe-selftest - INFO - oe-selftest - FAIL -
Required tests failed (successes=0, skipped=0, failures=0, errors=5)

On Thu, Apr 23, 2020 at 12:47 PM Ricardo Ribalda via
lists.openembedded.org
<ricardo.ribalda=gmail.com@lists.openembedded.org> wrote:
>
> On Thu, Apr 23, 2020 at 12:45 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Thu, 2020-04-23 at 11:57 +0200, Ricardo Ribalda Delgado wrote:
> > > Hi Richard
> > >
> > > Thanks for the test.
> > >
> > > All the errors seem to have the same pattern:
> > >
> > > FileNotFoundError: [Errno 2] No such file or directory:
> > > '/home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-
> > > 27135/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-
> > > r0/testimage/qemurunner_log.20200423015612'
> > >
> > > I do not believe that it is related to my patchset, I have seen that
> > > you are already running the selftest without the patchset
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/850
> > >
> > > Lets see if it also crashes or not. I am also trying to replicate
> > > locally.
> >
> > It did recur:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/862
> >
> > so your patches are probably not the cause, sorry! :)
>
> No problem, thanks!
>
> I am trying to replicate it anyway here. If I find a problem I will ping you.
>
> >
> > I'll continue to try and track it down.
> >
> > Cheers,
> >
> > Richard
> >
>
>
> --
> Ricardo Ribalda
> 



-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Richard Purdie]

On Thu, 2020-04-23 at 15:17 +0200, Ricardo Ribalda Delgado wrote:
> On my setup, using debian crops the error is even different :(
> 
> If you want I can keep trying, otherwise I will leave it to the
> experts ;)

Buried in there I think you have a setup problem:

Output from runqemu:
runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
may need to enable TUN (e.g. sudo modprobe tun)
runqemu - INFO - Cleaning up

I'm trying to narrow down the failing patch but its looking like it may
take me a while as its not immediately obvious :(

Cheers,

Richard

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

On Thu, Apr 23, 2020 at 3:23 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2020-04-23 at 15:17 +0200, Ricardo Ribalda Delgado wrote:
> > On my setup, using debian crops the error is even different :(
> >
> > If you want I can keep trying, otherwise I will leave it to the
> > experts ;)
>
> Buried in there I think you have a setup problem:

I launched it inside crops, I was expecting that everything was ok, but

/dev/net/tun is missing :(
pokyuser@e81a7801004c:/workdir/build$ ls -la /dev/net/tun
ls: cannot access '/dev/net/tun': No such file or directory


I can add it and retry, but it takes forever to run every test. Is
there a way to let it rerun, not starting from scratch every time?

2020-04-23 13:18:31,528 - oe-selftest - INFO - Adding layer libraries:
2020-04-23 13:18:31,529 - oe-selftest - INFO - /workdir/meta/lib
2020-04-23 13:18:31,529 - oe-selftest - INFO - /workdir/meta-yocto-bsp/lib
2020-04-23 13:18:31,529 - oe-selftest - INFO - /workdir/meta-selftest/lib
2020-04-23 13:18:31,531 - oe-selftest - INFO - Running bitbake -e to
test the configuration is valid/parsable
2020-04-23 13:18:34,955 - oe-selftest - ERROR - Build directory
/workdir/build-st already exists, aborting


Cheers
>
> Output from runqemu:
> runqemu - ERROR - TUN control device /dev/net/tun is unavailable; you
> may need to enable TUN (e.g. sudo modprobe tun)
> runqemu - INFO - Cleaning up
>
> I'm trying to narrow down the failing patch but its looking like it may
> take me a while as its not immediately obvious :(
>
> Cheers,
>
> Richard
>


-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Richard Purdie]

On Thu, 2020-04-23 at 15:17 +0200, Ricardo Ribalda Delgado wrote:
> On my setup, using debian crops the error is even different :(
> 
> If you want I can keep trying, otherwise I will leave it to the
> experts ;)

Just to update, I bisected this down to my sysroot patches in master-
next. Which makes no sense, how would sysroot changes cause wic to fail
in runqemu?

I've discovered that you have to run two tests in the same build one
after the other, specifically:

oe-selftest -r  devtool.DevtoolExtractTests.test_devtool_deploy_target wic.Wic2.test_qemu_efi

and I think my sysroot change "causes" problems as it adds a new test
class. This reorders tests and triggers this magic order problem. I
haven't confirmed but I think the above will fail without my patch.

The key is the wic test failing with:

FileNotFoundError: [Errno 2] No such file or directory: '/media/build1/poky/build-st/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-r0/testimage/qemurunner_log.20200425200658'

i.e. why is it writing to oe-selftest-image which is only used by the
devtool test?

The answer is I think the runqemu code is leaking logfile handles
somehow.

At least now we have a simpler and consistent reproducer and some idea
of the problem this should be easier to track down.

Cheers,

Richard

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Hi Richard

Thanks for the update!

On Sat, Apr 25, 2020 at 11:07 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Thu, 2020-04-23 at 15:17 +0200, Ricardo Ribalda Delgado wrote:
> > On my setup, using debian crops the error is even different :(
> >
> > If you want I can keep trying, otherwise I will leave it to the
> > experts ;)
>
> Just to update, I bisected this down to my sysroot patches in master-
> next. Which makes no sense, how would sysroot changes cause wic to fail
> in runqemu?
>
> I've discovered that you have to run two tests in the same build one
> after the other, specifically:
>
> oe-selftest -r  devtool.DevtoolExtractTests.test_devtool_deploy_target wic.Wic2.test_qemu_efi
>
> and I think my sysroot change "causes" problems as it adds a new test
> class. This reorders tests and triggers this magic order problem. I
> haven't confirmed but I think the above will fail without my patch.
>
> The key is the wic test failing with:
>
> FileNotFoundError: [Errno 2] No such file or directory: '/media/build1/poky/build-st/tmp/work/qemux86_64-poky-linux/oe-selftest-image/1.0-r0/testimage/qemurunner_log.20200425200658'
>
> i.e. why is it writing to oe-selftest-image which is only used by the
> devtool test?
>
> The answer is I think the runqemu code is leaking logfile handles
> somehow.
>
> At least now we have a simpler and consistent reproducer and some idea
> of the problem this should be easier to track down.

I have also been working on the issue, and eventhought I have not been
able to replicate locally the bug, the code insspection was pointing
to the same direction.
My running hypothesis is that the logger keeps writting to the
filehandler because nobody has removed it, so the second time we use
qemu, the log file is missing and all dies


Eg: Here I cannot see where the handler is removed
https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/targetcontrol.py?h=master-next#n122


Eg: Here I can

https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/utils/commands.py#n353


Unfortunately, as I said, I cannot replicate it, so it is a bit frustrating.

Since you can replicate it, could I send you a couple of patches, not
to fix it, but to find out if I am on the right direction?

Can you also publish a git branch somewhere with the current error, so
we work on the same codebase?

Or you just prefer to continue on your own?

Thanks

>
> Cheers,
>
> Richard
>
>
>


-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Richard Purdie]

On Sat, 2020-04-25 at 23:25 +0200, Ricardo Ribalda Delgado wrote:
> I have also been working on the issue, and eventhought I have not
> been
> able to replicate locally the bug, the code insspection was pointing
> to the same direction.
> My running hypothesis is that the logger keeps writting to the
> filehandler because nobody has removed it, so the second time we use
> qemu, the log file is missing and all dies
> 
> 
> Eg: Here I cannot see where the handler is removed
> https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/targetcontrol.py?h=master-next#n122
> 
> 
> Eg: Here I can
> 
> https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/utils/commands.py#n353
> 
> 
> Unfortunately, as I said, I cannot replicate it, so it is a bit
> frustrating.

The line I gave in the last email should reproduce it reliably. The key
is to have a runqemu session which you then clean the image for
(devtool does), then a write happens to a directory which no longer
exists from the logging.

> Since you can replicate it, could I send you a couple of patches, not
> to fix it, but to find out if I am on the right direction?

Since my other email I couldn't resist poking at the code. Locally this
patch fixed it so its now also running on the autobuilder:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=3e7d64adba65559e9b8af42e973c8524988b216a

> Can you also publish a git branch somewhere with the current error,
> so we work on the same codebase?
> 
> Or you just prefer to continue on your own?

Happy to have help but I'm simply using master-next and with any luck I
might have found it.

Cheers,

Richard

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Ricardo Ribalda]

Hi

On Sat, Apr 25, 2020 at 11:29 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sat, 2020-04-25 at 23:25 +0200, Ricardo Ribalda Delgado wrote:
> > I have also been working on the issue, and eventhought I have not
> > been
> > able to replicate locally the bug, the code insspection was pointing
> > to the same direction.
> > My running hypothesis is that the logger keeps writting to the
> > filehandler because nobody has removed it, so the second time we use
> > qemu, the log file is missing and all dies
> >
> >
> > Eg: Here I cannot see where the handler is removed
> > https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/targetcontrol.py?h=master-next#n122
> >
> >
> > Eg: Here I can
> >
> > https://git.openembedded.org/openembedded-core/tree/meta/lib/oeqa/utils/commands.py#n353
> >
> >
> > Unfortunately, as I said, I cannot replicate it, so it is a bit
> > frustrating.
>
> The line I gave in the last email should reproduce it reliably. The key
> is to have a runqemu session which you then clean the image for
> (devtool does), then a write happens to a directory which no longer
> exists from the logging.
>
> > Since you can replicate it, could I send you a couple of patches, not
> > to fix it, but to find out if I am on the right direction?
>
> Since my other email I couldn't resist poking at the code. Locally this
> patch fixed it so its now also running on the autobuilder:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=3e7d64adba65559e9b8af42e973c8524988b216a
>
> > Can you also publish a git branch somewhere with the current error,
> > so we work on the same codebase?
> >
> > Or you just prefer to continue on your own?
>
> Happy to have help but I'm simply using master-next and with any luck I
> might have found it.

If you could replicate it locally, and this fixes it. I am almost sure
you found it. I was looking in the same place ;)

Thanks

>
> Cheers,
>
> Richard
>
>
>


-- 
Ricardo Ribalda

Re: [OE-core] [PATCH v7 00/10] Fix permissions and embed-rotofs

[Richard Purdie]

On Sat, 2020-04-25 at 23:35 +0200, Ricardo Ribalda Delgado wrote:
> On Sat, Apr 25, 2020 at 11:29 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Sat, 2020-04-25 at 23:25 +0200, Ricardo Ribalda Delgado wrote:
> > Since my other email I couldn't resist poking at the code. Locally
> > this
> > patch fixed it so its now also running on the autobuilder:
> > 
> > http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=3e7d64adba65559e9b8af42e973c8524988b216a
> > 
> > > Can you also publish a git branch somewhere with the current
> > > error,
> > > so we work on the same codebase?
> > > 
> > > Or you just prefer to continue on your own?
> > 
> > Happy to have help but I'm simply using master-next and with any
> > luck I
> > might have found it.
> 
> If you could replicate it locally, and this fixes it. I am almost
> sure
> you found it. I was looking in the same place ;)

I think you'd have found it as you were on the right track and we came
to the same conclusion! Sorry about the confusion about where the fault
was, it was an unfortunate situation where it was really an underlying
bug which became exposed.

I'll get the logging fix sorted out and we can finally get some patches
merged. I'm a lot happier understanding how this was triggered.

Cheers,

Richard