[PATCH 0/5] systemd split networkd package

[PATCH 0/5] systemd split networkd package

[Adrian Freihofer]

First of all, this patch series fixes a bug in systemd.bbclass that
forces service files with an Also= relationship to be in the same
package. This may often be the case, but as long as the RDEPENDS
between the packages are set correctly, it is no problem to distribute
the service files over several packages.

If service files referenced with Also= are no longer automatically added
to the list of FILES, build errors caused by unpacked files are
expected. A bitbake world has shown that there are only two recipes in
oe-core that use automatic packing: busybox and dropbear. There are 2
patches that fix this.

Also= relation are probably most often used for sockets. Consequently
the systemd.bbclass should also allow to split socket files into
separate packages. However, that's proably practically not used and
would cause tons of packaging errors. Therefore the automatic packaging
feature for sockets is not changed.

The latest patch splits systemd-networkd into its own package. This adds
a little extra complexity, but splitting systemd into packages is a good
step towards making Yocto more usable in binary form.

Adrian Freihofer (5):
  busybox: add klogd service to FILES
  dropbear: add dropbearkey.service to FILES
  systemd.bbclass: refactor adding files
  systemd.bbclass: do not automatically add Also services
  systemd: split networkd into its own package

 meta/classes-recipe/systemd.bbclass           | 28 ++++++++-----------
 meta/recipes-core/busybox/busybox.inc         |  9 +++++-
 .../recipes-core/dropbear/dropbear_2024.86.bb |  2 ++
 meta/recipes-core/systemd/systemd_256.8.bb    | 27 ++++++++++++++++--
 4 files changed, 46 insertions(+), 20 deletions(-)

-- 
2.47.0

[PATCH 1/5] busybox: add klogd service to FILES

[Adrian Freihofer]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

busybox syslog.service file contains:
  Also=busybox-klogd.service

That's why busybox-klog.service got automatically added to
FILES:busybox-syslog. Since this incorrect auto adding feature gets
dropped from systemd.bbclass this needs to be added explicitly.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/recipes-core/busybox/busybox.inc | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
index 67a9111e5bd..f778cff0f4a 100644
--- a/meta/recipes-core/busybox/busybox.inc
+++ b/meta/recipes-core/busybox/busybox.inc
@@ -24,7 +24,14 @@ EXTRA_OEMAKE = "CC='${CC}' LD='${CCLD}' V=1 ARCH=${TARGET_ARCH} CROSS_COMPILE=${
 PACKAGES =+ "${PN}-httpd ${PN}-udhcpd ${PN}-udhcpc ${PN}-syslog ${PN}-mdev ${PN}-hwclock"
 
 FILES:${PN}-httpd = "${sysconfdir}/init.d/busybox-httpd /srv/www"
-FILES:${PN}-syslog = "${sysconfdir}/init.d/syslog* ${sysconfdir}/syslog-startup.conf* ${sysconfdir}/syslog.conf* ${systemd_system_unitdir}/syslog.service ${sysconfdir}/default/busybox-syslog"
+FILES:${PN}-syslog = "\
+    ${sysconfdir}/init.d/syslog* \
+    ${sysconfdir}/syslog-startup.conf* \
+    ${sysconfdir}/syslog.conf* \
+    ${systemd_system_unitdir}/syslog.service \
+    ${systemd_system_unitdir}/busybox-klogd.service \
+    ${sysconfdir}/default/busybox-syslog \
+"
 FILES:${PN}-mdev = "${sysconfdir}/init.d/mdev ${sysconfdir}/mdev.conf ${sysconfdir}/mdev/*"
 FILES:${PN}-udhcpd = "${sysconfdir}/init.d/busybox-udhcpd"
 FILES:${PN}-udhcpc = "${sysconfdir}/udhcpc.d ${datadir}/udhcpc"
-- 
2.47.0

[PATCH 2/5] dropbear: add dropbearkey.service to FILES

[Adrian Freihofer]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

dropbear.socket file contains:
  Also=dropbearkey.service

That's why dropbearkey.service got automatically added to FILES.
Since this incorrect auto adding feature gets dropped from
systemd.bbclass this needs to be added explicitly.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/recipes-core/dropbear/dropbear_2024.86.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-core/dropbear/dropbear_2024.86.bb b/meta/recipes-core/dropbear/dropbear_2024.86.bb
index e3f4d17896c..be246a0ccd5 100644
--- a/meta/recipes-core/dropbear/dropbear_2024.86.bb
+++ b/meta/recipes-core/dropbear/dropbear_2024.86.bb
@@ -129,3 +129,5 @@ pkg_postrm:${PN} () {
 }
 
 CONFFILES:${PN} = "${sysconfdir}/default/dropbear"
+
+FILES:${PN} += "${systemd_system_unitdir}/dropbearkey.service"
-- 
2.47.0

[PATCH 3/5] systemd.bbclass: refactor adding files

[Adrian Freihofer]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

The keys variable was intended as an array of keys. But it looks like
this has not been used for more than 10 years now. Adding files
automatically to packages needs probably anyway very specific code
rather than a generic loop. Lets simplify this a bit.
Using python code should also not be slower for these usually small
files.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/classes-recipe/systemd.bbclass | 28 ++++++++++++----------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-recipe/systemd.bbclass
index 7324af8555d..be77da48125 100644
--- a/meta/classes-recipe/systemd.bbclass
+++ b/meta/classes-recipe/systemd.bbclass
@@ -124,29 +124,26 @@ python systemd_populate_packages() {
         return appended
 
     # Add systemd files to FILES:*-systemd, parse for Also= and follow recursive
-    def systemd_add_files_and_parse(pkg_systemd, path, service, keys):
+    def systemd_add_files_and_parse(pkg_systemd, path, service):
         # avoid infinite recursion
         if systemd_append_file(pkg_systemd, oe.path.join(path, service)):
             fullpath = oe.path.join(d.getVar("D"), path, service)
             if service.find('.service') != -1:
                 # for *.service add *@.service
                 service_base = service.replace('.service', '')
-                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys)
+                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service')
             if service.find('.socket') != -1:
                 # for *.socket add *.service and *@.service
                 service_base = service.replace('.socket', '')
-                systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys)
-                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys)
-            for key in keys.split():
-                # recurse all dependencies found in keys ('Also';'Conflicts';..) and add to files
-                cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" % (key, shlex.quote(fullpath), key)
-                pipe = os.popen(cmd, 'r')
-                line = pipe.readline()
-                while line:
-                    line = line.replace('\n', '')
-                    systemd_add_files_and_parse(pkg_systemd, path, line, keys)
-                    line = pipe.readline()
-                pipe.close()
+                systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service')
+                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service')
+            # Add all units which have an Also= referring a unit in this package to this package as well.
+            with open(fullpath, 'r') as unit_f:
+                for line in unit_f:
+                    if line.startswith('Also'):
+                        also_unit = line.split('=', 1)[1].strip()
+                        bb.warn("also: %s" % also_unit)
+                        systemd_add_files_and_parse(pkg_systemd, path, also_unit)
 
     # Check service-files and call systemd_add_files_and_parse for each entry
     def systemd_check_services():
@@ -155,7 +152,6 @@ python systemd_populate_packages() {
         searchpaths.append(d.getVar("systemd_user_unitdir"))
         systemd_packages = d.getVar('SYSTEMD_PACKAGES')
 
-        keys = 'Also'
         # scan for all in SYSTEMD_SERVICE[]
         for pkg_systemd in systemd_packages.split():
             for service in get_package_var(d, 'SYSTEMD_SERVICE', pkg_systemd).split():
@@ -179,7 +175,7 @@ python systemd_populate_packages() {
                             break
 
                 if path_found != '':
-                    systemd_add_files_and_parse(pkg_systemd, path_found, service, keys)
+                    systemd_add_files_and_parse(pkg_systemd, path_found, service)
                 else:
                     bb.fatal("Didn't find service unit '{0}', specified in SYSTEMD_SERVICE:{1}. {2}".format(
                         service, pkg_systemd, "Also looked for service unit '{0}'.".format(base) if base is not None else ""))
-- 
2.47.0

[PATCH 4/5] systemd.bbclass: do not automatically add Also services

[Adrian Freihofer]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

Starting a systemd service “Also” does not mean that both services
must be in the same package. However, the systemd.bbclass enforces
this.

Example:
  a.service:
    [Install]
    Also=b.service
If a.service is packed in package A, b.service is automatically packed
into package A as well. This happens even if b.service is explicitly
added to package B using FILES and SYSTEMD_SERVICE variables.

The automatic packing of socket files with the corresponding service
files is probably a widely used feature of systemd.bbclass. This bahavior
does not change.

Adding regular service files to a package just because it is another
service in the same package that "Also" uses the service is a bug that
this commit fixes.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/classes-recipe/systemd.bbclass | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-recipe/systemd.bbclass
index be77da48125..4b4470b7b34 100644
--- a/meta/classes-recipe/systemd.bbclass
+++ b/meta/classes-recipe/systemd.bbclass
@@ -132,18 +132,18 @@ python systemd_populate_packages() {
                 # for *.service add *@.service
                 service_base = service.replace('.service', '')
                 systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service')
+                # Add the socket unit which is referred by the Also= in this service file to the same package.
+                with open(fullpath, 'r') as unit_f:
+                    for line in unit_f:
+                        if line.startswith('Also'):
+                            also_unit = line.split('=', 1)[1].strip()
+                            if also_unit.find('.socket') != -1:
+                                systemd_add_files_and_parse(pkg_systemd, path, also_unit)
             if service.find('.socket') != -1:
                 # for *.socket add *.service and *@.service
                 service_base = service.replace('.socket', '')
                 systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service')
                 systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service')
-            # Add all units which have an Also= referring a unit in this package to this package as well.
-            with open(fullpath, 'r') as unit_f:
-                for line in unit_f:
-                    if line.startswith('Also'):
-                        also_unit = line.split('=', 1)[1].strip()
-                        bb.warn("also: %s" % also_unit)
-                        systemd_add_files_and_parse(pkg_systemd, path, also_unit)
 
     # Check service-files and call systemd_add_files_and_parse for each entry
     def systemd_check_services():
-- 
2.47.0

[PATCH 5/5] systemd: split networkd into its own package

[Adrian Freihofer]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

Support of images with different network management implementations
without having to recompile systemd and other components.

Fedora does this as well since systemd version 246.6-2:
https://src.fedoraproject.org/rpms/systemd/blob/rawhide/f/systemd.spec#_578
This proves that it is technically possible and officially supported by
systemd.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 meta/recipes-core/systemd/systemd_256.8.bb | 27 +++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_256.8.bb b/meta/recipes-core/systemd/systemd_256.8.bb
index c1915d7e941..ca43c8d2ddb 100644
--- a/meta/recipes-core/systemd/systemd_256.8.bb
+++ b/meta/recipes-core/systemd/systemd_256.8.bb
@@ -412,6 +412,7 @@ PACKAGE_BEFORE_PN = "\
     ${PN}-journal-remote \
     ${PN}-kernel-install \
     ${PN}-mime \
+    ${PN}-networkd \
     ${PN}-rpm-macros \
     ${PN}-udev-rules \
     ${PN}-vconsole-setup \
@@ -440,6 +441,7 @@ SYSTEMD_PACKAGES = "${@bb.utils.contains('PACKAGECONFIG', 'binfmt', '${PN}-binfm
                     ${@bb.utils.contains('PACKAGECONFIG', 'microhttpd', '${PN}-journal-gatewayd', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'microhttpd', '${PN}-journal-remote', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'journal-upload', '${PN}-journal-upload', '', d)} \
+                    ${@bb.utils.contains('PACKAGECONFIG', 'networkd', '${PN}-networkd', '', d)} \
 "
 SYSTEMD_SERVICE:${PN}-binfmt = "systemd-binfmt.service"
 
@@ -453,7 +455,7 @@ GROUPADD_PARAM:${PN} = "-r systemd-journal;"
 GROUPADD_PARAM:udev = "-r render"
 GROUPADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'polkit_hostnamed_fallback', '-r systemd-hostname;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'coredump', '--system -d / -M --shell /sbin/nologin systemd-coredump;', '', d)}"
-USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'networkd', '--system -d / -M --shell /sbin/nologin systemd-network;', '', d)}"
+USERADD_PARAM:${PN}-networkd = "--system -d / -M --shell /sbin/nologin systemd-network"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'polkit', '--system --no-create-home --user-group --home-dir ${datadir}/polkit-1 polkitd;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'resolved', '--system -d / -M --shell /sbin/nologin systemd-resolve;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'timesyncd', '--system -d / -M --shell /sbin/nologin systemd-timesync;', '', d)}"
@@ -620,6 +622,27 @@ FILES:${PN}-extra-utils = "\
 FILES:${PN}-mime = "${MIMEDIR}"
 RRECOMMENDS:${PN} += "${PN}-mime"
 
+FILES:${PN}-networkd = "\
+    ${bindir}/networkctl \
+    ${datadir}/dbus-1/system-services/org.freedesktop.network1.service \
+    ${datadir}/dbus-1/system.d/org.freedesktop.network1.conf \
+    ${datadir}/polkit-1/actions/org.freedesktop.network1.policy \
+    ${nonarch_libdir}/sysusers.d/systemd-network.conf \
+    ${nonarch_libdir}/tmpfiles.d/systemd-network.conf \
+    ${sysconfdir}/systemd/networkd.conf \
+    ${systemd_system_unitdir}/systemd-networkd* \
+    ${systemd_unitdir}/network/*.network \
+    ${systemd_unitdir}/network/*.network.example \
+    ${systemd_unitdir}/networkd.conf \
+    ${systemd_unitdir}/systemd-networkd* \
+"
+# systemd-networkd-persistent-storage.service BindsTo=systemd-networkd.service
+# systemd-networkd.service has Also=systemd-networkd-wait-online.service
+SYSTEMD_SERVICE:${PN}-networkd = "systemd-networkd.service"
+CONFFILES:${PN}-networkd = "${sysconfdir}/systemd/networkd.conf"
+RDEPENDS:${PN}-networkd += "${PN}"
+RRECOMMENDS:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'networkd', '${PN}-networkd', '', d)}"
+
 FILES:${PN}-udev-rules = "\
                         ${nonarch_libdir}/udev/rules.d/70-uaccess.rules \
                         ${nonarch_libdir}/udev/rules.d/71-seat.rules \
@@ -630,7 +653,6 @@ FILES:${PN}-udev-rules = "\
 CONFFILES:${PN} = "${sysconfdir}/systemd/coredump.conf \
 	${sysconfdir}/systemd/journald.conf \
 	${sysconfdir}/systemd/logind.conf \
-	${sysconfdir}/systemd/networkd.conf \
 	${sysconfdir}/systemd/pstore.conf \
 	${sysconfdir}/systemd/resolved.conf \
 	${sysconfdir}/systemd/sleep.conf \
@@ -698,7 +720,6 @@ FILES:${PN} = " ${base_bindir}/* \
                 ${nonarch_libdir}/modprobe.d/README \
                 ${datadir}/dbus-1/system.d/org.freedesktop.timedate1.conf \
                 ${datadir}/dbus-1/system.d/org.freedesktop.locale1.conf \
-                ${datadir}/dbus-1/system.d/org.freedesktop.network1.conf \
                 ${datadir}/dbus-1/system.d/org.freedesktop.resolve1.conf \
                 ${datadir}/dbus-1/system.d/org.freedesktop.systemd1.conf \
                 ${@bb.utils.contains('PACKAGECONFIG', 'polkit_hostnamed_fallback', '${datadir}/dbus-1/system.d/org.freedesktop.hostname1_no_polkit.conf', '', d)} \
-- 
2.47.0

Re: [OE-core] [PATCH 5/5] systemd: split networkd into its own package

[Mathieu Dubois-Briand]

On Wed Dec 11, 2024 at 10:58 PM CET, Adrian Freihofer via lists.openembedded.org wrote:
> From: Adrian Freihofer <adrian.freihofer@siemens.com>
>
> Support of images with different network management implementations
> without having to recompile systemd and other components.
>
> Fedora does this as well since systemd version 246.6-2:
> https://src.fedoraproject.org/rpms/systemd/blob/rawhide/f/systemd.spec#_578
> This proves that it is technically possible and officially supported by
> systemd.
>
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---

Hi Adrian,

I believe this patch or another one from this series is responsible of
some warnings we can see on the autobuilder:

stdio: WARNING: core-image-full-cmdline-1.0-r0 do_rootfs: User systemd-network has never been defined
stdio: WARNING: core-image-sato-1.0-r0 do_rootfs: User systemd-network has never been defined
stdio: WARNING: core-image-sato-sdk-1.0-r0 do_rootfs: User systemd-network has never been defined

https://valkyrie.yoctoproject.org/#/builders/10/builds/638/steps/11/logs/warnings
https://valkyrie.yoctoproject.org/#/builders/64/builds/559/steps/12/logs/warnings
https://valkyrie.yoctoproject.org/#/builders/13/builds/561/steps/12/logs/warnings

Can you look at this issue please ?

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com