[PATCH 0/6] Pull request (cover letter only)

[PATCH 0/6] Pull request (cover letter only)

[Vyacheslav Yurkov]

This is a V1 of overlayfs-etc image feature implementation, that allows
to setup the whole /etc under overlayfs. Please review and merge if you
find it OK

The following changes since commit 0d15632f3db787d3f08eb260732567e62f52ffb3:

  libtasn1: upgrade 4.17.0 -> 4.18.0 (2021-11-16 22:19:47 +0000)

are available in the Git repository at:

  git://github.com/UVV-gh/openembedded-core feature/overlayfs-etc
  https://github.com/UVV-gh/openembedded-core/tree/feature/overlayfs-etc

Vyacheslav Yurkov (6):
  overlayfs-etc: mount etc as overlayfs
  wic: image for overlayfs-etc tests
  image: add overlayfs-etc image feature
  oeqa/selftest: overlayfs helper function
  oeqa/selftest: unit tests for overlayfs-etc
  overlayfs: update notes on /etc

 meta-selftest/wic/overlayfs_etc.wks.in    |   4 +
 meta/classes/image.bbclass                |   3 +-
 meta/classes/overlayfs-etc.bbclass        |  93 ++++++++++++
 meta/classes/overlayfs.bbclass            |   1 +
 meta/lib/oeqa/selftest/cases/overlayfs.py | 173 ++++++++++++++++++++--
 5 files changed, 257 insertions(+), 17 deletions(-)
 create mode 100644 meta-selftest/wic/overlayfs_etc.wks.in
 create mode 100644 meta/classes/overlayfs-etc.bbclass

-- 
2.28.0

[PATCH 1/6] overlayfs-etc: mount etc as overlayfs

[Vyacheslav Yurkov]

This class provides an image feature that mounts /etc as an overlayfs
file system. This is an extension for existing overlayfs class, which
doesn't support /etc

Signed-off-by: Alfred Schapansky <alfred.schapansky@avantys.de>
Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/overlayfs-etc.bbclass | 93 ++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)
 create mode 100644 meta/classes/overlayfs-etc.bbclass

diff --git a/meta/classes/overlayfs-etc.bbclass b/meta/classes/overlayfs-etc.bbclass
new file mode 100644
index 0000000000..78caf0211b
--- /dev/null
+++ b/meta/classes/overlayfs-etc.bbclass
@@ -0,0 +1,93 @@
+# Class for setting up /etc in overlayfs
+#
+# In order to have /etc directory in overlayfs a special handling at early boot stage is required
+# The idea is to supply a custom init script that mounts /etc before launching actual init program,
+# because the latter already requires /etc to be mounted
+#
+# The configuration must be machine specific. You should at least set these two variables if you
+# are not happy with default values:
+#   OVERLAYFS_ETC_MOUNT_POINT ?= "/data"
+#   OVERLAYFS_ETC_DEVICE ?= "/dev/mmcblk0p2"
+#
+# To control more mount options you should consider also setting file system type and mount options:
+#   OVERLAYFS_ETC_FSTYPE ?= "ext4"
+#   OVERLAYFS_ETC_MOUNT_OPTIONS ?= "defaults"
+#
+# The class provides two options for /sbin/init generation
+# 1. Default option is to rename original /sbin/init to /sbin/init.orig and place generated init under
+#    original name, i.e. /sbin/init. It has an advantage that you won't need to change any kernel
+#    parameters in order to make it work, but it poses a restriction that package-management can't
+#    be used, becaause updating init manager would remove generated script
+# 2. If you are would like to keep original init as is, you can set
+#    OVERLAYFS_ETC_USE_ORIG_INIT_NAME = "0"
+#    Then generated init will be named /sbin/preinit and you would need to extend you kernel parameters
+#    manually in your bootloader configuration.
+#
+# Regardless which mode you choose, update and migration strategy of configuration files under /etc
+# overlay is out of scope of this class
+
+ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "overlayfs-etc", "create_overlayfs_etc_preinit;", "", d)}'
+IMAGE_FEATURES_CONFLICTS_overlayfs-etc = "package-management"
+
+OVERLAYFS_ETC_MOUNT_POINT ?= "/data"
+OVERLAYFS_ETC_FSTYPE ?= "ext4"
+OVERLAYFS_ETC_DEVICE ?= "/dev/mmcblk0p2"
+OVERLAYFS_ETC_USE_ORIG_INIT_NAME ?= "1"
+OVERLAYFS_ETC_MOUNT_OPTIONS ?= "defaults"
+
+python create_overlayfs_etc_preinit() {
+    PreinitTemplate = """#!/bin/sh
+
+echo "PREINIT: Start"
+
+PATH=/sbin:/bin:/usr/sbin:/usr/bin
+mount -o remount,rw /
+
+mkdir -p /proc
+mkdir -p /sys
+mkdir -p /run
+mkdir -p /var/run
+
+mount -t proc proc /proc
+mount -t sysfs sysfs /sys
+
+[ -z "$CONSOLE" ] && CONSOLE="/dev/console"
+
+mkdir -p {OVERLAYFS_ETC_MOUNT_POINT}
+if mount -n -t {OVERLAYFS_ETC_FSTYPE} -o {OVERLAYFS_ETC_MOUNT_OPTIONS} {OVERLAYFS_ETC_DEVICE} {OVERLAYFS_ETC_MOUNT_POINT}
+then
+    mkdir -p {OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/upper
+    mkdir -p {OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/work
+    mount -n -t overlay -o upperdir={OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/upper,lowerdir=/etc,workdir={OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/work {OVERLAYFS_ETC_MOUNT_POINT}/overlay-etc/upper /etc || echo "PREINIT: Mounting etc-overlay failed!"
+else
+    echo "PREINIT: Mounting </data> failed!"
+fi
+
+echo "PREINIT: done; starting </sbin/init>"
+exec {SBIN_INIT_NAME}
+"""
+
+    useOrigInit = oe.types.boolean(d.getVar('OVERLAYFS_ETC_USE_ORIG_INIT_NAME'))
+    preinitPath = oe.path.join(d.getVar("IMAGE_ROOTFS"), d.getVar("base_sbindir"), "preinit")
+    initBaseName = oe.path.join(d.getVar("base_sbindir"), "init")
+    origInitNameSuffix = ".orig"
+
+    args = {
+        'OVERLAYFS_ETC_MOUNT_POINT': d.getVar('OVERLAYFS_ETC_MOUNT_POINT'),
+        'OVERLAYFS_ETC_MOUNT_OPTIONS': d.getVar('OVERLAYFS_ETC_MOUNT_OPTIONS'),
+        'OVERLAYFS_ETC_FSTYPE': d.getVar('OVERLAYFS_ETC_FSTYPE'),
+        'OVERLAYFS_ETC_DEVICE': d.getVar('OVERLAYFS_ETC_DEVICE'),
+        'SBIN_INIT_NAME': initBaseName + origInitNameSuffix if useOrigInit else initBaseName
+    }
+
+    if useOrigInit:
+        # rename original /sbin/init
+        origInit = oe.path.join(d.getVar("IMAGE_ROOTFS"), initBaseName)
+        bb.debug(1, "rootfs path %s, init path %s, test %s" % (d.getVar('IMAGE_ROOTFS'), origInit, d.getVar("IMAGE_ROOTFS")))
+        bb.utils.rename(origInit, origInit + origInitNameSuffix)
+        preinitPath = origInit
+
+    with open(preinitPath, 'w') as f:
+        f.write(PreinitTemplate.format(**args))
+    os.chmod(preinitPath, 0o755)
+}
-- 
2.28.0

[PATCH 2/6] wic: image for overlayfs-etc tests

[Vyacheslav Yurkov]

Introduce wic image for overlayfs-etc tests with a dedicated /data
partition and configurable kernel parameters

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta-selftest/wic/overlayfs_etc.wks.in | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 meta-selftest/wic/overlayfs_etc.wks.in

diff --git a/meta-selftest/wic/overlayfs_etc.wks.in b/meta-selftest/wic/overlayfs_etc.wks.in
new file mode 100644
index 0000000000..1e1e5836e7
--- /dev/null
+++ b/meta-selftest/wic/overlayfs_etc.wks.in
@@ -0,0 +1,4 @@
+part /boot --active --source bootimg-biosplusefi --ondisk sda --sourceparams="loader=grub-efi" --align 1024
+part / --source rootfs --ondisk sda --fstype=ext4 --use-uuid --align 1024
+part --ondisk sda --fstype=ext4 --size=5 --align 1024
+bootloader --ptable gpt --timeout=1 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0 ${OVERLAYFS_INIT_OPTION}"
-- 
2.28.0

[PATCH 3/6] image: add overlayfs-etc image feature

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/image.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6c759fdf70..e198f041b1 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -15,6 +15,7 @@ IMGCLASSES += "${@bb.utils.contains('IMAGE_FSTYPES', 'container', 'image-contain
 IMGCLASSES += "image_types_wic"
 IMGCLASSES += "rootfs-postcommands"
 IMGCLASSES += "image-postinst-intercepts"
+IMGCLASSES += "overlayfs-etc"
 inherit ${IMGCLASSES}
 
 TOOLCHAIN_TARGET_TASK += "${PACKAGE_INSTALL}"
@@ -33,7 +34,7 @@ INHIBIT_DEFAULT_DEPS = "1"
 # IMAGE_FEATURES may contain any available package group
 IMAGE_FEATURES ?= ""
 IMAGE_FEATURES[type] = "list"
-IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging"
+IMAGE_FEATURES[validitems] += "debug-tweaks read-only-rootfs read-only-rootfs-delayed-postinsts stateless-rootfs empty-root-password allow-empty-password allow-root-login post-install-logging overlayfs-etc"
 
 # Generate companion debugfs?
 IMAGE_GEN_DEBUGFS ?= "0"
-- 
2.28.0

[PATCH 4/6] oeqa/selftest: overlayfs helper function

[Vyacheslav Yurkov]

Move helper functions out of class scope so they can be used in other tests

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/lib/oeqa/selftest/cases/overlayfs.py | 32 +++++++++++------------
 1 file changed, 16 insertions(+), 16 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py
index 84242a1605..43415778ce 100644
--- a/meta/lib/oeqa/selftest/cases/overlayfs.py
+++ b/meta/lib/oeqa/selftest/cases/overlayfs.py
@@ -5,16 +5,16 @@
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
 
-class OverlayFSTests(OESelftestTestCase):
-    """Overlayfs class usage tests"""
+def getline_qemu(out, line):
+    for l in out.split('\n'):
+        if line in l:
+            return l
 
-    def getline_qemu(self, out, line):
-        for l in out.split('\n'):
-            if line in l:
-                return l
+def getline(res, line):
+    return getline_qemu(res.output, line)
 
-    def getline(self, res, line):
-        return self.getline_qemu(res.output, line)
+class OverlayFSTests(OESelftestTestCase):
+    """Overlayfs class usage tests"""
 
     def add_overlay_conf_to_machine(self):
         machine_inc = """
@@ -40,7 +40,7 @@ inherit overlayfs
         self.write_recipeinc('overlayfs-user', overlayfs_recipe_append)
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "overlayfs-user was skipped: missing required distro features")
+        line = getline(res, "overlayfs-user was skipped: missing required distro features")
         self.assertTrue("overlayfs" in res.output, msg=res.output)
         self.assertTrue("systemd" in res.output, msg=res.output)
         self.assertTrue("ERROR: Required build target 'core-image-minimal' has no buildable providers." in res.output, msg=res.output)
@@ -61,9 +61,9 @@ DISTRO_FEATURES += "systemd overlayfs"
         self.add_overlay_conf_to_machine()
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "Unit name mnt-overlay.mount not found in systemd unit directories")
+        line = getline(res, "Unit name mnt-overlay.mount not found in systemd unit directories")
         self.assertTrue(line and line.startswith("WARNING:"), msg=res.output)
-        line = self.getline(res, "Not all mount units are installed by the BSP")
+        line = getline(res, "Not all mount units are installed by the BSP")
         self.assertTrue(line and line.startswith("ERROR:"), msg=res.output)
 
     def test_mount_unit_not_set(self):
@@ -81,7 +81,7 @@ DISTRO_FEATURES += "systemd overlayfs"
         self.write_config(config)
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration")
+        line = getline(res, "A recipe uses overlayfs class but there is no OVERLAYFS_MOUNT_POINT set in your MACHINE configuration")
         self.assertTrue(line and line.startswith("Parsing recipes...ERROR:"), msg=res.output)
 
     def test_wrong_mount_unit_set(self):
@@ -104,7 +104,7 @@ OVERLAYFS_MOUNT_POINT[usr-share-overlay] = "/usr/share/overlay"
         self.set_machine_config(wrong_machine_config)
 
         res = bitbake('core-image-minimal', ignore_status=True)
-        line = self.getline(res, "Missing required mount point for OVERLAYFS_MOUNT_POINT[mnt-overlay] in your MACHINE configuration")
+        line = getline(res, "Missing required mount point for OVERLAYFS_MOUNT_POINT[mnt-overlay] in your MACHINE configuration")
         self.assertTrue(line and line.startswith("Parsing recipes...ERROR:"), msg=res.output)
 
     def test_correct_image(self):
@@ -201,11 +201,11 @@ EOT
             # /usr/share/my-application as an overlay (see overlayfs-user recipe)
             status, output = qemu.run_serial("/bin/mount -t tmpfs,overlay")
 
-            line = self.getline_qemu(output, "on /mnt/overlay")
+            line = getline_qemu(output, "on /mnt/overlay")
             self.assertTrue(line and line.startswith("tmpfs"), msg=output)
 
-            line = self.getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/my-application")
+            line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/my-application")
             self.assertTrue(line and line.startswith("overlay"), msg=output)
 
-            line = self.getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
+            line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
             self.assertTrue(line and line.startswith("overlay"), msg=output)
-- 
2.28.0

[PATCH 5/6] oeqa/selftest: unit tests for overlayfs-etc

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/lib/oeqa/selftest/cases/overlayfs.py | 141 ++++++++++++++++++++++
 1 file changed, 141 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/overlayfs.py b/meta/lib/oeqa/selftest/cases/overlayfs.py
index 43415778ce..4623215a47 100644
--- a/meta/lib/oeqa/selftest/cases/overlayfs.py
+++ b/meta/lib/oeqa/selftest/cases/overlayfs.py
@@ -209,3 +209,144 @@ EOT
 
             line = getline_qemu(output, "upperdir=/mnt/overlay/upper/usr/share/another-overlay-mount")
             self.assertTrue(line and line.startswith("overlay"), msg=output)
+
+class OverlayFSEtcRunTimeTests(OESelftestTestCase):
+    """overlayfs-etc class tests"""
+
+    def test_image_feature_conflict(self):
+        """
+        Summary:   Overlayfs-etc is not allowed to be used with package-management
+        Expected:  Feature conflict
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+EXTRA_IMAGE_FEATURES += "package-management"
+"""
+
+        self.write_config(config)
+
+        res = bitbake('core-image-minimal', ignore_status=True)
+        line = getline(res, "contains conflicting IMAGE_FEATURES")
+        self.assertTrue("overlayfs-etc" in res.output, msg=res.output)
+        self.assertTrue("package-management" in res.output, msg=res.output)
+
+    def test_image_feature_is_missing_class_included(self):
+        configAppend = """
+INHERIT += "overlayfs-etc"
+"""
+        self.run_check_image_feature(configAppend)
+
+    def test_image_feature_is_missing(self):
+        self.run_check_image_feature()
+
+    def run_check_image_feature(self, appendToConfig=""):
+        """
+        Summary:   Overlayfs-etc class is not applied when image feature is not set
+                   even if we inherit it directly,
+        Expected:  Image is created successfully but /etc is not an overlay
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = f"""
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+IMAGE_FSTYPES += "wic"
+WKS_FILE = "overlayfs_etc.wks.in"
+
+EXTRA_IMAGE_FEATURES += "read-only-rootfs"
+# Image configuration for overlayfs-etc
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+OVERLAYFS_ETC_DEVICE = "/dev/sda3"
+{appendToConfig}
+"""
+
+        self.write_config(config)
+
+        bitbake('core-image-minimal')
+
+        with runqemu('core-image-minimal', image_fstype='wic') as qemu:
+            status, output = qemu.run_serial("/bin/mount")
+
+            line = getline_qemu(output, "upperdir=/data/overlay-etc/upper")
+            self.assertFalse(line, msg=output)
+
+    def test_sbin_init_preinit(self):
+        self.run_sbin_init(False)
+
+    def test_sbin_init_original(self):
+        self.run_sbin_init(True)
+
+    def run_sbin_init(self, origInit):
+        """
+        Summary:   Confirm we can replace original init and mount overlay on top of /etc
+        Expected:  Image is created successfully and /etc is mounted as an overlay
+        Author:    Vyacheslav Yurkov <uvv.mail@gmail.com>
+        """
+
+        config = """
+DISTRO_FEATURES += "systemd"
+
+# Use systemd as init manager
+VIRTUAL-RUNTIME_init_manager = "systemd"
+
+# enable overlayfs in the kernel
+KERNEL_EXTRA_FEATURES:append = " features/overlayfs/overlayfs.scc"
+
+IMAGE_FSTYPES += "wic"
+OVERLAYFS_INIT_OPTION = "{OVERLAYFS_INIT_OPTION}"
+WKS_FILE = "overlayfs_etc.wks.in"
+
+EXTRA_IMAGE_FEATURES += "read-only-rootfs"
+# Image configuration for overlayfs-etc
+EXTRA_IMAGE_FEATURES += "overlayfs-etc"
+IMAGE_FEATURES:remove = "package-management"
+OVERLAYFS_ETC_MOUNT_POINT = "/data"
+OVERLAYFS_ETC_DEVICE = "/dev/sda3"
+OVERLAYFS_ETC_USE_ORIG_INIT_NAME = "{OVERLAYFS_ETC_USE_ORIG_INIT_NAME}"
+"""
+
+        args = {
+            'OVERLAYFS_INIT_OPTION': "" if origInit else "init=/sbin/preinit",
+            'OVERLAYFS_ETC_USE_ORIG_INIT_NAME': int(origInit == True)
+        }
+
+        self.write_config(config.format(**args))
+
+        bitbake('core-image-minimal')
+        testFile = "/etc/my-test-data"
+
+        with runqemu('core-image-minimal', image_fstype='wic', discard_writes=False) as qemu:
+            status, output = qemu.run_serial("/bin/mount")
+
+            line = getline_qemu(output, "/dev/sda3")
+            self.assertTrue("/data" in output, msg=output)
+
+            line = getline_qemu(output, "upperdir=/data/overlay-etc/upper")
+            self.assertTrue(line and line.startswith("/data/overlay-etc/upper on /etc type overlay"), msg=output)
+
+            status, output = qemu.run_serial("touch " + testFile)
+            status, output = qemu.run_serial("sync")
+            status, output = qemu.run_serial("ls -1 " + testFile)
+            line = getline_qemu(output, testFile)
+            self.assertTrue(line and line.startswith(testFile), msg=output)
+
+        # Check that file exists in /etc after reboot
+        with runqemu('core-image-minimal', image_fstype='wic') as qemu:
+            status, output = qemu.run_serial("ls -1 " + testFile)
+            line = getline_qemu(output, testFile)
+            self.assertTrue(line and line.startswith(testFile), msg=output)
-- 
2.28.0

[PATCH 6/6] overlayfs: update notes on /etc

[Vyacheslav Yurkov]

Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
---
 meta/classes/overlayfs.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/overlayfs.bbclass b/meta/classes/overlayfs.bbclass
index 3c0f4dc882..f1b8086ea8 100644
--- a/meta/classes/overlayfs.bbclass
+++ b/meta/classes/overlayfs.bbclass
@@ -31,6 +31,7 @@
 #   OVERLAYFS_WRITABLE_PATHS[mnt-overlay] = "/usr/share/another-application"
 #
 # Note: the class does not support /etc directory itself, because systemd depends on it
+# For /etc directory use overlayfs-etc class
 
 REQUIRED_DISTRO_FEATURES += "systemd overlayfs"
 
-- 
2.28.0

Re: [OE-core] [PATCH 1/6] overlayfs-etc: mount etc as overlayfs

[Ross Burton]

On Fri, 19 Nov 2021 at 06:15, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
> +# Class for setting up /etc in overlayfs
> +#
> +# In order to have /etc directory in overlayfs a special handling at early boot stage is required
> +# The idea is to supply a custom init script that mounts /etc before launching actual init program,
> +# because the latter already requires /etc to be mounted

Can you elaborate on the exact scenario that is being supported here?
Is it read-only / with /etc (and potentially more) in an overlayfs?

> +OVERLAYFS_ETC_MOUNT_POINT ?= "/data"
> +OVERLAYFS_ETC_FSTYPE ?= "ext4"
> +OVERLAYFS_ETC_DEVICE ?= "/dev/mmcblk0p2"

I'm guessing these are not universal defaults, but the values that you use.

If you can't provide default values which will actually work out of
the box, just set them to "" and the class can abort if they're not
set.

Ross

Re: [OE-core] [PATCH 1/6] overlayfs-etc: mount etc as overlayfs

[Richard Purdie]

On Fri, 2021-11-19 at 07:15 +0100, Vyacheslav Yurkov wrote:
> This class provides an image feature that mounts /etc as an overlayfs
> file system. This is an extension for existing overlayfs class, which
> doesn't support /etc
> 
> Signed-off-by: Alfred Schapansky <alfred.schapansky@avantys.de>
> Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
> ---
>  meta/classes/overlayfs-etc.bbclass | 93 ++++++++++++++++++++++++++++++
>  1 file changed, 93 insertions(+)
>  create mode 100644 meta/classes/overlayfs-etc.bbclass
> 
> diff --git a/meta/classes/overlayfs-etc.bbclass b/meta/classes/overlayfs-etc.bbclass
> new file mode 100644
> index 0000000000..78caf0211b
> --- /dev/null
> +++ b/meta/classes/overlayfs-etc.bbclass
> @@ -0,0 +1,93 @@
> +# Class for setting up /etc in overlayfs
> +#
> +# In order to have /etc directory in overlayfs a special handling at early boot stage is required
> +# The idea is to supply a custom init script that mounts /etc before launching actual init program,
> +# because the latter already requires /etc to be mounted
> +#
> +# The configuration must be machine specific. You should at least set these two variables if you
> +# are not happy with default values:
> +#   OVERLAYFS_ETC_MOUNT_POINT ?= "/data"
> +#   OVERLAYFS_ETC_DEVICE ?= "/dev/mmcblk0p2"
> +#
> +# To control more mount options you should consider also setting file system type and mount options:
> +#   OVERLAYFS_ETC_FSTYPE ?= "ext4"
> +#   OVERLAYFS_ETC_MOUNT_OPTIONS ?= "defaults"
> +#
> +# The class provides two options for /sbin/init generation
> +# 1. Default option is to rename original /sbin/init to /sbin/init.orig and place generated init under
> +#    original name, i.e. /sbin/init. It has an advantage that you won't need to change any kernel
> +#    parameters in order to make it work, but it poses a restriction that package-management can't
> +#    be used, becaause updating init manager would remove generated script
> +# 2. If you are would like to keep original init as is, you can set
> +#    OVERLAYFS_ETC_USE_ORIG_INIT_NAME = "0"
> +#    Then generated init will be named /sbin/preinit and you would need to extend you kernel parameters
> +#    manually in your bootloader configuration.
> +#
> +# Regardless which mode you choose, update and migration strategy of configuration files under /etc
> +# overlay is out of scope of this class
> +
> +ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "overlayfs-etc", "create_overlayfs_etc_preinit;", "", d)}'
> +IMAGE_FEATURES_CONFLICTS_overlayfs-etc = "package-management"
> +
> +OVERLAYFS_ETC_MOUNT_POINT ?= "/data"
> +OVERLAYFS_ETC_FSTYPE ?= "ext4"
> +OVERLAYFS_ETC_DEVICE ?= "/dev/mmcblk0p2"
> +OVERLAYFS_ETC_USE_ORIG_INIT_NAME ?= "1"
> +OVERLAYFS_ETC_MOUNT_OPTIONS ?= "defaults"
> +
> +python create_overlayfs_etc_preinit() {
> +    PreinitTemplate = """#!/bin/sh
> +
> +echo "PREINIT: Start"
> +
> +PATH=/sbin:/bin:/usr/sbin:/usr/bin
> +mount -o remount,rw /
> +
> +mkdir -p /proc
> +mkdir -p /sys
> +mkdir -p /run

Sorry about the delay replying on this. Something has been bothering me about
the code and I haven't been able to articulate it. I think Ross is right about
part of it, the defaults are very device specific and need to be invalid by
default which checks to ensure they're set.

The second thing which I don't really like is this mix of shell and python, it
is very confusing to read. I'd much rather we put template files into
meta/lib/files/ and then read them from there for the variable substitutions. We
do this for other scripts. I appreciate the overlayfs class merged without that
but I think we need to fix this there as well.

Adding the tests in this series is really great through, thanks for that!

Cheers,

Richard

Re: [OE-core] [PATCH 1/6] overlayfs-etc: mount etc as overlayfs

[Vyacheslav Yurkov]

On 09.12.2021 11:59, Ross Burton wrote:
> On Fri, 19 Nov 2021 at 06:15, Vyacheslav Yurkov <uvv.mail@gmail.com> wrote:
>> +# Class for setting up /etc in overlayfs
>> +#
>> +# In order to have /etc directory in overlayfs a special handling at early boot stage is required
>> +# The idea is to supply a custom init script that mounts /etc before launching actual init program,
>> +# because the latter already requires /etc to be mounted
> Can you elaborate on the exact scenario that is being supported here?
> Is it read-only / with /etc (and potentially more) in an overlayfs?

Correct, that's a read-only / with only /etc in an overlayfs. For other 
directories (not under /etc) there's a systemd based solution in 
overlayfs.bbclass.

>> +OVERLAYFS_ETC_MOUNT_POINT ?= "/data"
>> +OVERLAYFS_ETC_FSTYPE ?= "ext4"
>> +OVERLAYFS_ETC_DEVICE ?= "/dev/mmcblk0p2"
> I'm guessing these are not universal defaults, but the values that you use.
>
> If you can't provide default values which will actually work out of
> the box, just set them to "" and the class can abort if they're not
> set.
>
> Ross

Thanks for the feedback. Will fix that.

Vyacheslav

Re: [OE-core] [PATCH 1/6] overlayfs-etc: mount etc as overlayfs

[Vyacheslav Yurkov]

On 09.12.2021 12:08, Richard Purdie wrote:
> The second thing which I don't really like is this mix of shell and python, it
> is very confusing to read. I'd much rather we put template files into
> meta/lib/files/ and then read them from there for the variable substitutions. We
> do this for other scripts. I appreciate the overlayfs class merged without that
> but I think we need to fix this there as well.

Thanks for the feedback, Richard.
I see only meta/files. I guess you meant this directory.

Will do the same for overlayfs.bbclass too.

> Adding the tests in this series is really great through, thanks for that!
>
> Cheers,
>
> Richard

Regards,
Vyacheslav

Re: [OE-core] [PATCH 1/6] overlayfs-etc: mount etc as overlayfs

[Richard Purdie]

On Thu, 2021-12-09 at 16:19 +0100, Vyacheslav Yurkov wrote:
> On 09.12.2021 12:08, Richard Purdie wrote:
> > The second thing which I don't really like is this mix of shell and python, it
> > is very confusing to read. I'd much rather we put template files into
> > meta/lib/files/ and then read them from there for the variable substitutions. We
> > do this for other scripts. I appreciate the overlayfs class merged without that
> > but I think we need to fix this there as well.
> 
> Thanks for the feedback, Richard.
> I see only meta/files. I guess you meant this directory.

I did, yes. Not sure where the lib came from! :)

> Will do the same for overlayfs.bbclass too.

Thanks!

Cheers,

Richard