[PATCH 0/3] useradd fixes, tests and documentation

[PATCH 0/3] useradd fixes, tests and documentation

[Eilís 'pidge' Ní Fhlannagáin]

This patch series fixes and/or tests for some issues in useradd dependencies. It
adds a new variable "USERADD_DEPENDS" that is required now for recipes that have
dependencies on other recipes useradd.

Fixes/Tests:
Bug 13419 - recipes that add users to groups cannot rely on other recipes creating those groups (when population from sstate happens)
Bug 13904 - do_prepare_recipe_sysroot: postinst-useradd-* does not run in order of dependency and sometimes fails
Bug 13279 - Make sure users/groups exist for package_write_* tasks
Bug 15084 - For some reason using of same user in two recipes does not work properly

Tests:
Bug 14961 - addtask between do_populate_sysroot and do_package breaks useradd class

Documentation for useradd is also included.

Eilís 'pidge' Ní Fhlannagáin (3):
  useradd: Fix issues with useradd dependencies
  usergrouptests.py: Add test for switching between static-ids
  variables.rst: Add USERADD_DEPENDS

 documentation/ref-manual/variables.rst        | 15 +++++
 .../selftest-users/creategroup1.bb            | 32 +++++++++++
 .../selftest-users/creategroup2.bb            | 33 +++++++++++
 .../selftest-users/useraddbadtask.bb          | 20 +++++++
 meta/classes/useradd.bbclass                  |  4 +-
 .../lib/oeqa/selftest/cases/usergrouptests.py | 55 +++++++++++++++++++
 6 files changed, 158 insertions(+), 1 deletion(-)
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup1.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup2.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/useraddbadtask.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/usergrouptests.py

-- 
2.34.1

[PATCH 1/3] useradd: Fix issues with useradd dependencies

[Eilís 'pidge' Ní Fhlannagáin]

If recipe A requires the useradd actions of recipe B we need to
ensure that recipe B is part of the recipe A dependancy chain. In
order to do that, we introduce USERADD_DEPENDS. This makes sure
that the do_populate_sysroot_setscene of recipe B exists for
recipe A in case of a missing TMPDIR. This requires changes made in
runqueue.py by RP.

This commit along with the runqueue fixes effects:
Bug 13419 - recipes that add users to groups cannot rely on other recipes creating those groups (when population from sstate happens)
Bug 13904 - do_prepare_recipe_sysroot: postinst-useradd-* does not run in order of dependency and sometimes fails
Bug 13279 - Make sure users/groups exist for package_write_* tasks
Bug 15084 - For some reason using of same user in two recipes does not work properly

I've included the start of self-testing for useradd by adding tests for
13419 (which ends up testing 13904, 13279, 15084 by virtue of them all
      having the same root cause)

I've also added (and disabled), a test for 14961 - addtask between do_populate_sysroot and do_package breaks useradd class. A fix is still needed for this, but that fix is a better error message that stops people from doing this.

Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <pidge@baylibre.com>
---
 .../selftest-users/creategroup1.bb            | 32 ++++++++++++++++++
 .../selftest-users/creategroup2.bb            | 33 +++++++++++++++++++
 .../selftest-users/useraddbadtask.bb          | 20 +++++++++++
 meta/classes/useradd.bbclass                  |  4 ++-
 .../lib/oeqa/selftest/cases/usergrouptests.py | 26 +++++++++++++++
 5 files changed, 114 insertions(+), 1 deletion(-)
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup1.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/creategroup2.bb
 create mode 100644 meta-selftest/recipes-test/selftest-users/useraddbadtask.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/usergrouptests.py

diff --git a/meta-selftest/recipes-test/selftest-users/creategroup1.bb b/meta-selftest/recipes-test/selftest-users/creategroup1.bb
new file mode 100644
index 00000000000..ebbbfaa83d3
--- /dev/null
+++ b/meta-selftest/recipes-test/selftest-users/creategroup1.bb
@@ -0,0 +1,32 @@
+SUMMARY = "creategroup pt 1"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+LICENSE = "MIT"
+
+DEPENDS = "coreutils-native"
+
+S = "${WORKDIR}"
+
+inherit useradd allarch
+
+USERADD_PACKAGES = "${PN}"
+USERADD_PARAM:${PN} = "-u 5555 --gid grouptest gt1"
+GROUPADD_PARAM:${PN} = "-r grouptest"
+
+TESTDIR = "${D}${sysconfdir}/creategroup"
+
+do_install() {
+	install -d   ${TESTDIR}
+	install -d   ${TESTDIR}/dir
+	touch        ${TESTDIR}/file
+	ln -s ./file ${TESTDIR}/symlink
+	install -d   ${TESTDIR}/fifotest
+	mkfifo       ${TESTDIR}/fifotest/fifo
+
+	chown    gt1:grouptest ${TESTDIR}/file
+	chown -R gt1:grouptest ${TESTDIR}/dir
+	chown -h gt1:grouptest ${TESTDIR}/symlink
+	chown -R gt1:grouptest ${TESTDIR}/fifotest
+}
+
+FILES:${PN} = "${sysconfdir}/creategroup/*"
diff --git a/meta-selftest/recipes-test/selftest-users/creategroup2.bb b/meta-selftest/recipes-test/selftest-users/creategroup2.bb
new file mode 100644
index 00000000000..ef697f09b4d
--- /dev/null
+++ b/meta-selftest/recipes-test/selftest-users/creategroup2.bb
@@ -0,0 +1,33 @@
+SUMMARY = "creategroup pt 2"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+LICENSE = "MIT"
+
+DEPENDS = "coreutils-native"
+USERADD_DEPENDS = "creategroup1"
+
+S = "${WORKDIR}"
+
+inherit useradd allarch
+
+USERADD_PACKAGES = "${PN}"
+USERADD_PARAM:${PN} = "-u 5556 --gid grouptest gt2"
+
+TESTDIR = "${D}${sysconfdir}/creategroup"
+
+do_install() {
+	install -d   ${TESTDIR}
+	install -d   ${TESTDIR}/dir
+	touch        ${TESTDIR}/file
+	ln -s ./file ${TESTDIR}/symlink
+	install -d   ${TESTDIR}/fifotest
+	mkfifo       ${TESTDIR}/fifotest/fifo
+
+	chown    gt2:grouptest ${TESTDIR}/file
+	chown -R gt2:grouptest ${TESTDIR}/dir
+	chown -h gt2:grouptest ${TESTDIR}/symlink
+	chown -R gt2:grouptest ${TESTDIR}/fifotest
+}
+
+FILES:${PN} = "${sysconfdir}/creategroup/*"
+
diff --git a/meta-selftest/recipes-test/selftest-users/useraddbadtask.bb b/meta-selftest/recipes-test/selftest-users/useraddbadtask.bb
new file mode 100644
index 00000000000..99e04a80b34
--- /dev/null
+++ b/meta-selftest/recipes-test/selftest-users/useraddbadtask.bb
@@ -0,0 +1,20 @@
+SUMMARY = "UserAddBadTask"
+LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
+
+LICENSE = "MIT"
+
+DEPENDS:append = "coreutils-native"
+
+S = "${WORKDIR}"
+
+inherit useradd allarch
+
+USERADD_PACKAGES = "${PN}"
+USERADD_PARAM:${PN} = "-u 5555 --gid groupaddtask useraddtask"
+GROUPADD_PARAM:${PN} = "-r groupaddtask"
+
+do_badthingshappen() {
+ echo "foo"
+}
+
+addtask badthingshappen after do_populate_sysroot before do_package
diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
index 4d3bd9a5f56..a4b8a2d6d60 100644
--- a/meta/classes/useradd.bbclass
+++ b/meta/classes/useradd.bbclass
@@ -177,9 +177,11 @@ SYSROOT_PREPROCESS_FUNCS += "${SYSROOTFUNC}"
 
 SSTATEPREINSTFUNCS:append:class-target = " useradd_sysroot_sstate"
 
+USERADD_DEPENDS ??= ""
+DEPENDS += "${USERADD_DEPENDS}"
 do_package_setscene[depends] += "${USERADDSETSCENEDEPS}"
 do_populate_sysroot_setscene[depends] += "${USERADDSETSCENEDEPS}"
-USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
+USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"
 USERADDSETSCENEDEPS = ""
 
 # Recipe parse-time sanity checks
diff --git a/meta/lib/oeqa/selftest/cases/usergrouptests.py b/meta/lib/oeqa/selftest/cases/usergrouptests.py
new file mode 100644
index 00000000000..a63fce54dc9
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/usergrouptests.py
@@ -0,0 +1,26 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, get_bb_var
+import bb.utils
+import os
+
+class UserGroupTests(OESelftestTestCase):
+    def test_group_from_dep_package(self):
+        self.logger.info("Building creategroup2")
+        bitbake(' creategroup2 creategroup1')
+        bitbake(' creategroup2 creategroup1 -c clean')
+        self.logger.info("Packaging creategroup2")
+        self.assertTrue(bitbake(' creategroup2 -c package'))
+
+    def _test_add_task_between_p_sysroot_and_package(self):
+        self.logger.info("Cleaning sstate for useraddbadtask")
+        #bitbake(' useraddbadtask -f -c cleansstate')
+        self.logger.info("Building useraddbadtask")
+        # This is expected to fail due to bug #14961
+        self.assertTrue(bitbake(' useraddbadtask -C fetch'))
+    
-- 
2.34.1

[PATCH 2/3] usergrouptests.py: Add test for switching between static-ids

[Eilís 'pidge' Ní Fhlannagáin]

This test is related to https://bugzilla.yoctoproject.org/show_bug.cgi?id=12107

At the moment it doesn't seem to be able to actually replicate this
issue in the bug, which tells me it's likely fixed.

Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <pidge@baylibre.com>
---
 .../lib/oeqa/selftest/cases/usergrouptests.py | 45 +++++++++++++++----
 1 file changed, 37 insertions(+), 8 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/usergrouptests.py b/meta/lib/oeqa/selftest/cases/usergrouptests.py
index a63fce54dc9..c74b5678d00 100644
--- a/meta/lib/oeqa/selftest/cases/usergrouptests.py
+++ b/meta/lib/oeqa/selftest/cases/usergrouptests.py
@@ -5,11 +5,15 @@
 #
 
 from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, get_bb_var
-import bb.utils
+from oeqa.utils.commands import bitbake, get_bb_var, get_test_layer
 import os
 
 class UserGroupTests(OESelftestTestCase):
+    def _test_add_task_between_p_sysroot_and_package(self):
+        self.logger.info("Building useraddbadtask")
+        # This is expected to fail due to bug #14961
+        self.assertTrue(bitbake(' useraddbadtask -C fetch'))
+
     def test_group_from_dep_package(self):
         self.logger.info("Building creategroup2")
         bitbake(' creategroup2 creategroup1')
@@ -17,10 +21,35 @@ class UserGroupTests(OESelftestTestCase):
         self.logger.info("Packaging creategroup2")
         self.assertTrue(bitbake(' creategroup2 -c package'))
 
-    def _test_add_task_between_p_sysroot_and_package(self):
-        self.logger.info("Cleaning sstate for useraddbadtask")
-        #bitbake(' useraddbadtask -f -c cleansstate')
-        self.logger.info("Building useraddbadtask")
-        # This is expected to fail due to bug #14961
-        self.assertTrue(bitbake(' useraddbadtask -C fetch'))
+    def test_static_useradd_from_dynamic(self):
+        metaselftestpath = get_test_layer()
+        self.logger.info("Building core-image-minimal to generate passwd/group file")
+        bitbake(' core-image-minimal')
+        self.logger.info("Setting up useradd-staticids")
+        repropassdir = os.path.join(metaselftestpath, "conf/include")
+        os.makedirs(repropassdir)
+        etcdir=os.path.join(os.path.join(os.path.join(get_bb_var("TMPDIR"), "work"), \
+                            os.path.join(get_bb_var("MACHINE").replace("-","_")+"-poky-linux", "core-image-minimal/1.0/rootfs/etc")))
+        import shutil
+        shutil.copy(os.path.join(etcdir, "passwd"), os.path.join(repropassdir, "reproducable-passwd"))
+        shutil.copy(os.path.join(etcdir, "group"), os.path.join(repropassdir, "reproducable-group"))
+        # Copy the original local.conf
+        shutil.copyfile(os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf'), os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf.orig'))
+
+        self.write_config("USERADDEXTENSION = \"useradd-staticids\"")
+        self.write_config("USERADD_ERROR_DYNAMIC ??= \"error\"")
+        self.write_config("USERADD_UID_TABLES += \"conf/include/reproducible-passwd\"")
+        self.write_config("USERADD_GID_TABLES += \"conf/include/reproducible-group\"")
+        self.logger.info("Rebuild with staticids")
+        bitbake(' core-image-minimal')
+        shutil.copyfile(os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf.orig'), os.path.join(os.environ.get('BUILDDIR'), 'conf/local.conf'))
+        self.logger.info("Rebuild without staticids")
+        bitbake(' core-image-minimal')
+        self.write_config("USERADDEXTENSION = \"useradd-staticids\"")
+        self.write_config("USERADD_ERROR_DYNAMIC ??= \"error\"")
+        self.write_config("USERADD_UID_TABLES += \"files/static-passwd\"")
+        self.write_config("USERADD_GID_TABLES += \"files/static-group\"")
+        self.logger.info("Rebuild with other staticids")
+        self.assertTrue(bitbake(' core-image-minimal'))
+    
     
-- 
2.34.1

[PATCH 3/3] variables.rst: Add USERADD_DEPENDS

[Eilís 'pidge' Ní Fhlannagáin]

Adding USERADD_DEPENDS to the variables document

Signed-off-by: Eilís 'pidge' Ní Fhlannagáin <pidge@baylibre.com>
---
 documentation/ref-manual/variables.rst | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index b394d310990..9ec0ce4b4ca 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -9736,6 +9736,21 @@ system and gives an overview of their function and contents.
       ``meta-poky/conf/templates/default/local.conf.sample`` in the
       :term:`Source Directory`.
 
+   :term:`USERADD_DEPENDS`
+      Lists a recipe's USERADD dependencies. These are dependencies on
+      other recipes whose create users and groups via useradd.bbclass and are
+      needed by the recipe at build time. This variable is now required for
+      recipes that rely on other recipes for part of their useradd.
+      
+      As an example, consider a recipe ``foo``. Recipe ``bar`` creates a group
+      that foo requires for it's useradd tasks. In order to ensure that the
+      useradd tasks exist for foo, we add the the following assignment::
+
+          $USERADD_DEPENDS = "bar"
+
+      The practical effect of this assignment is that bar's sysroot_setscene is
+      added to :term:`USERADDSETSCENEDEPS` variable.
+
    :term:`USERADD_ERROR_DYNAMIC`
       If set to ``error``, forces the OpenEmbedded build system to produce
       an error if the user identification (``uid``) and group
-- 
2.34.1

Re: [OE-core] [PATCH 1/3] useradd: Fix issues with useradd dependencies

[Alexander Kanavin]

On Thu, 7 Dec 2023 at 13:45, Eilís 'pidge' Ní Fhlannagáin
<pidge@baylibre.com> wrote:
> -USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
> +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"

I'd like to clarify: do all these recent fixes mean that we can also
add dynamic library dependencies of shadow-native into this list (it
becomes an issue with latest versions of shadow currently not in
oe-core)?

Specifically, this commit

https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/package-version-updates&id=9fdbe321fa53785cc35bd1bb4054366383ef20e5

doing

-USERADDSETSCENEDEPS:class-target =
"${MLPREFIX}base-passwd:do_populate_sysroot_setscene
pseudo-native:do_populate_sysroot_setscene
shadow-native:do_populate_sysroot_setscene
${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
+USERADDSETSCENEDEPS:class-target =
"${MLPREFIX}base-passwd:do_populate_sysroot_setscene \
+ pseudo-native:do_populate_sysroot_setscene \
+ shadow-native:do_populate_sysroot_setscene \
+ attr-native:do_populate_sysroot_setscene \
+ libbsd-native:do_populate_sysroot_setscene \
+ libmd-native:do_populate_sysroot_setscene \
+ ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"

has been rejected by RP with the suggestion that shadow should be
linked statically, but now I'm not sure if the argument still holds.

Alex

Re: [OE-core] [PATCH 1/3] useradd: Fix issues with useradd dependencies

[Richard Purdie]

On Thu, 2023-12-07 at 13:58 +0100, Alexander Kanavin wrote:
> On Thu, 7 Dec 2023 at 13:45, Eilís 'pidge' Ní Fhlannagáin
> <pidge@baylibre.com> wrote:
> > -USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
> > +USERADDSETSCENEDEPS:class-target = "${MLPREFIX}base-passwd:do_populate_sysroot_setscene pseudo-native:do_populate_sysroot_setscene shadow-native:do_populate_sysroot_setscene ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene ${@' '.join(['%s:do_populate_sysroot_setscene' % pkg for pkg in d.getVar("USERADD_DEPENDS").split()])}"
> 
> I'd like to clarify: do all these recent fixes mean that we can also
> add dynamic library dependencies of shadow-native into this list (it
> becomes an issue with latest versions of shadow currently not in
> oe-core)?
> 
> Specifically, this commit
> 
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/package-version-updates&id=9fdbe321fa53785cc35bd1bb4054366383ef20e5
> 
> doing
> 
> -USERADDSETSCENEDEPS:class-target =
> "${MLPREFIX}base-passwd:do_populate_sysroot_setscene
> pseudo-native:do_populate_sysroot_setscene
> shadow-native:do_populate_sysroot_setscene
> ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
> +USERADDSETSCENEDEPS:class-target =
> "${MLPREFIX}base-passwd:do_populate_sysroot_setscene \
> + pseudo-native:do_populate_sysroot_setscene \
> + shadow-native:do_populate_sysroot_setscene \
> + attr-native:do_populate_sysroot_setscene \
> + libbsd-native:do_populate_sysroot_setscene \
> + libmd-native:do_populate_sysroot_setscene \
> + ${MLPREFIX}shadow-sysroot:do_populate_sysroot_setscene"
> 
> has been rejected by RP with the suggestion that shadow should be
> linked statically, but now I'm not sure if the argument still holds.

After my recent change, we could in theory do this as bitbake might be
able to make it work. It does run the risk of circular dependencies and
these setscene inter-task dependencies are *horrible* in general from
an sstate perspective though as my other comments about these
dependencies leading to poor behaviour still stand.

To be clear about what you're saying above, you're saying that if any
of those native dependencies are missing, we cannot use/install the
sstate object. We have to defer installing that sstate until all the
others are downloaded and available too.

My preference is still to statically link shadow to avoid this pretty
horrible side effect.

Cheers,

Richard

Re: [PATCH 1/3] useradd: Fix issues with useradd dependencies

[Jermain Horsman]

> If recipe A requires the useradd actions of recipe B we need to
> ensure that recipe B is part of the recipe A dependancy chain. In
> order to do that, we introduce USERADD_DEPENDS. This makes sure
> that the do_populate_sysroot_setscene of recipe B exists for
> recipe A in case of a missing TMPDIR. This requires changes made in
> runqueue.py by RP.

I was wondering if all required changes including these would be considered suitable to backport to e.g. kirkstone?
Or am I going to have to look forward to scarthgap for these :)

Sincerely,

Jermain