[OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup

[OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup

[Alex Kiernan]

When copying the package database files for the debugfs, add individual
file copy as well as tree copying. After the debug rootfs has been
created, cleanup the package files.

This then allows us to avoid a problem where (for rpm at least)
extraneous files in the debug rootfs would cause failures during
oe-selftest because some files existed in both regular and debugfs
images.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

Changes in v2:
- New in v2

 meta/lib/oe/rootfs.py | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 890ba5f03984..1a48ed10b3f6 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta):
     def _cleanup(self):
         pass
 
-    def _setup_dbg_rootfs(self, dirs):
+    def _setup_dbg_rootfs(self, package_paths):
         gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0'
         if gen_debugfs != '1':
            return
@@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta):
         bb.utils.mkdirhier(self.image_rootfs)
 
         bb.note("  Copying back package database...")
-        for dir in dirs:
-            if not os.path.isdir(self.image_rootfs + '-orig' + dir):
-                continue
-            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir))
-            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True)
+        for path in package_paths:
+            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
+            if os.path.isdir(self.image_rootfs + '-orig' + path):
+                shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True)
+            elif os.path.isfile(self.image_rootfs + '-orig' + path):
+                shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path)
 
         # Copy files located in /usr/lib/debug or /usr/src/debug
         for dir in ["/usr/lib/debug", "/usr/src/debug"]:
@@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta):
             bb.note("  Install extra debug packages...")
             self.pm.install(extra_debug_pkgs.split(), True)
 
+        bb.note("  Removing package database...")
+        for path in package_paths:
+            if os.path.isdir(self.image_rootfs + path):
+                shutil.rmtree(self.image_rootfs + path)
+            elif os.path.isfile(self.image_rootfs + path):
+                os.remove(self.image_rootfs + path)
+
         bb.note("  Rename debug rootfs...")
         try:
             shutil.rmtree(self.image_rootfs + '-dbg')
-- 
2.39.0

[OE-Core][PATCH v2 2/3] rpm: Pick debugfs package db files/dirs explicitly

[Alex Kiernan]

Rather than copying the entire /etc hierarchy, specify the pieces we
actually need.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

Changes in v2:
- New in v2

 meta/lib/oe/package_manager/rpm/rootfs.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager/rpm/rootfs.py b/meta/lib/oe/package_manager/rpm/rootfs.py
index d4c415f68cad..3ba539632063 100644
--- a/meta/lib/oe/package_manager/rpm/rootfs.py
+++ b/meta/lib/oe/package_manager/rpm/rootfs.py
@@ -110,7 +110,7 @@ class PkgRootfs(Rootfs):
         if self.progress_reporter:
             self.progress_reporter.next_stage()
 
-        self._setup_dbg_rootfs(['/etc', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
+        self._setup_dbg_rootfs(['/etc/rpm', '/etc/rpmrc', '/etc/dnf', '/var/lib/rpm', '/var/cache/dnf', '/var/lib/dnf'])
 
         execute_pre_post_process(self.d, rpm_post_process_cmds)
 
-- 
2.39.0

[OE-Core][PATCH v2 3/3] eudev: Add group sgx to eudev package

[Alex Kiernan]

Fix startup warning:

  udevd[171]: specified group 'sgx' unknown

This mirrors the change in bab455cd9b1b ("systemd: add group sgx to udev
package") for systemd-udev.

Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

Changes in v2:
- Rework rpm handling so that the debugfs doesn't include a spurious
  copy of /etc which caused the gdbserver selftest to fail because of a
  duplicate /etc/gshadow file

 meta/recipes-core/udev/eudev_3.2.12.bb | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/udev/eudev_3.2.12.bb b/meta/recipes-core/udev/eudev_3.2.12.bb
index 572ccecafd0c..4268bcc2c5de 100644
--- a/meta/recipes-core/udev/eudev_3.2.12.bb
+++ b/meta/recipes-core/udev/eudev_3.2.12.bb
@@ -18,7 +18,7 @@ SRC_URI[sha256sum] = "ccdd64ec3c381d3c3ed0e99d2e70d1f62988c7763de89ca7bdffafa5ea
 
 GITHUB_BASE_URI = "https://github.com/eudev-project/eudev/releases"
 
-inherit autotools update-rc.d qemu pkgconfig features_check manpages github-releases
+inherit autotools update-rc.d qemu pkgconfig features_check manpages github-releases useradd
 
 CONFLICT_DISTRO_FEATURES = "systemd"
 
@@ -85,3 +85,6 @@ pkg_postinst:${PN}-hwdb () {
 pkg_prerm:${PN}-hwdb () {
 	rm -f $D${sysconfdir}/udev/hwdb.bin
 }
+
+USERADD_PACKAGES = "${PN}"
+GROUPADD_PARAM:${PN} = "-r sgx"
-- 
2.39.0

RE: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Alex Kiernan
> Sent: den 20 juli 2023 12:20
> To: openembedded-core@lists.openembedded.org
> Cc: Alex Kiernan <alex.kiernan@gmail.com>
> Subject: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup
> 
> When copying the package database files for the debugfs, add individual
> file copy as well as tree copying. After the debug rootfs has been
> created, cleanup the package files.
> 
> This then allows us to avoid a problem where (for rpm at least)
> extraneous files in the debug rootfs would cause failures during
> oe-selftest because some files existed in both regular and debugfs
> images.
> 
> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
> ---
> 
> Changes in v2:
> - New in v2
> 
>  meta/lib/oe/rootfs.py | 20 ++++++++++++++------
>  1 file changed, 14 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> index 890ba5f03984..1a48ed10b3f6 100644
> --- a/meta/lib/oe/rootfs.py
> +++ b/meta/lib/oe/rootfs.py
> @@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta):
>      def _cleanup(self):
>          pass
> 
> -    def _setup_dbg_rootfs(self, dirs):
> +    def _setup_dbg_rootfs(self, package_paths):
>          gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0'
>          if gen_debugfs != '1':
>             return
> @@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta):
>          bb.utils.mkdirhier(self.image_rootfs)
> 
>          bb.note("  Copying back package database...")
> -        for dir in dirs:
> -            if not os.path.isdir(self.image_rootfs + '-orig' + dir):
> -                continue
> -            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir))
> -            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True)
> +        for path in package_paths:
> +            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
> +            if os.path.isdir(self.image_rootfs + '-orig' + path):
> +                shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True)
> +            elif os.path.isfile(self.image_rootfs + '-orig' + path):
> +                shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path)
> 
>          # Copy files located in /usr/lib/debug or /usr/src/debug
>          for dir in ["/usr/lib/debug", "/usr/src/debug"]:
> @@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta):
>              bb.note("  Install extra debug packages...")
>              self.pm.install(extra_debug_pkgs.split(), True)
> 
> +        bb.note("  Removing package database...")
> +        for path in package_paths:
> +            if os.path.isdir(self.image_rootfs + path):
> +                shutil.rmtree(self.image_rootfs + path)
> +            elif os.path.isfile(self.image_rootfs + path):
> +                os.remove(self.image_rootfs + path)

What's the reason to do it like this rather than calling
self.pm.remove_packaging_data()?

I also just noticed that we apparently have a local change that does the 
above, but where we also do:

     # Remove /etc as it may clash with rootfs.  Also saves some space.
     bb.utils.remove(oe.path.join(self.image_rootfs, 'etc'), True)

I do not know if that would be appropriate to do here as well?

> +
>          bb.note("  Rename debug rootfs...")
>          try:
>              shutil.rmtree(self.image_rootfs + '-dbg')
> --
> 2.39.0

//Peter

Re: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup

[Alex Kiernan]

On Mon, Aug 7, 2023 at 9:21 PM Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Alex Kiernan
> > Sent: den 20 juli 2023 12:20
> > To: openembedded-core@lists.openembedded.org
> > Cc: Alex Kiernan <alex.kiernan@gmail.com>
> > Subject: [OE-Core][PATCH v2 1/3] rootfs: Add debugfs package db file copy and cleanup
> >
> > When copying the package database files for the debugfs, add individual
> > file copy as well as tree copying. After the debug rootfs has been
> > created, cleanup the package files.
> >
> > This then allows us to avoid a problem where (for rpm at least)
> > extraneous files in the debug rootfs would cause failures during
> > oe-selftest because some files existed in both regular and debugfs
> > images.
> >
> > Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
> > ---
> >
> > Changes in v2:
> > - New in v2
> >
> >  meta/lib/oe/rootfs.py | 20 ++++++++++++++------
> >  1 file changed, 14 insertions(+), 6 deletions(-)
> >
> > diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
> > index 890ba5f03984..1a48ed10b3f6 100644
> > --- a/meta/lib/oe/rootfs.py
> > +++ b/meta/lib/oe/rootfs.py
> > @@ -106,7 +106,7 @@ class Rootfs(object, metaclass=ABCMeta):
> >      def _cleanup(self):
> >          pass
> >
> > -    def _setup_dbg_rootfs(self, dirs):
> > +    def _setup_dbg_rootfs(self, package_paths):
> >          gen_debugfs = self.d.getVar('IMAGE_GEN_DEBUGFS') or '0'
> >          if gen_debugfs != '1':
> >             return
> > @@ -122,11 +122,12 @@ class Rootfs(object, metaclass=ABCMeta):
> >          bb.utils.mkdirhier(self.image_rootfs)
> >
> >          bb.note("  Copying back package database...")
> > -        for dir in dirs:
> > -            if not os.path.isdir(self.image_rootfs + '-orig' + dir):
> > -                continue
> > -            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(dir))
> > -            shutil.copytree(self.image_rootfs + '-orig' + dir, self.image_rootfs + dir, symlinks=True)
> > +        for path in package_paths:
> > +            bb.utils.mkdirhier(self.image_rootfs + os.path.dirname(path))
> > +            if os.path.isdir(self.image_rootfs + '-orig' + path):
> > +                shutil.copytree(self.image_rootfs + '-orig' + path, self.image_rootfs + path, symlinks=True)
> > +            elif os.path.isfile(self.image_rootfs + '-orig' + path):
> > +                shutil.copyfile(self.image_rootfs + '-orig' + path, self.image_rootfs + path)
> >
> >          # Copy files located in /usr/lib/debug or /usr/src/debug
> >          for dir in ["/usr/lib/debug", "/usr/src/debug"]:
> > @@ -162,6 +163,13 @@ class Rootfs(object, metaclass=ABCMeta):
> >              bb.note("  Install extra debug packages...")
> >              self.pm.install(extra_debug_pkgs.split(), True)
> >
> > +        bb.note("  Removing package database...")
> > +        for path in package_paths:
> > +            if os.path.isdir(self.image_rootfs + path):
> > +                shutil.rmtree(self.image_rootfs + path)
> > +            elif os.path.isfile(self.image_rootfs + path):
> > +                os.remove(self.image_rootfs + path)
>
> What's the reason to do it like this rather than calling
> self.pm.remove_packaging_data()?
>

Just looking now, I suspect only that I didn't notice that it existed!
Though looking across the other packaging implementations, it looks
like you'd get different things removed than were passed initially.
Certainly looks like a refactor would be in order.

> I also just noticed that we apparently have a local change that does the
> above, but where we also do:
>
>      # Remove /etc as it may clash with rootfs.  Also saves some space.
>      bb.utils.remove(oe.path.join(self.image_rootfs, 'etc'), True)
>
> I do not know if that would be appropriate to do here as well?
>

The clash was exactly what I was fixing here (gshadow), but prior to
these two changes you got the whole of /etc copied, so a clash was
exactly what you were in danger of running into - anything that
clashes now, I think wants investigating as to why, rather than just
wiping out `/etc` entirely.

-- 
Alex Kiernan