[OE-core][dunfell 0/5] Patch review

[OE-core][dunfell 0/5] Patch review

[Steve Sakoman]

Please review this set of changes for dunfell and have comments back by 
end of day Friday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5460

The following changes since commit 96798603273bbab40a7e19bbb67a9a5398648c21:

  openssh: Move sshdgenkeys.service to sshd.socket (2023-06-07 05:49:27 -1000)

are available in the Git repository at:

  https://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Martin Siegumfeldt (1):
  systemd-systemctl: fix instance template WantedBy symlink construction

Nikhil R (1):
  libwebp: Fix CVE-2023-1999

Qiu Tingting (1):
  e2fsprogs: fix ptest bug for second running

Randy MacLeod (1):
  vim: upgrade 9.0.1429 -> 9.0.1527

Richard Purdie (1):
  selftest/reproducible: Allow native/cross reuse in test

 meta/lib/oeqa/selftest/cases/reproducible.py  |  4 +-
 .../systemd/systemd-systemctl/systemctl       | 13 ++++-
 .../e2fsprogs/e2fsprogs/run-ptest             |  1 +
 .../e2fsprogs/e2fsprogs_1.45.7.bb             |  3 +
 .../webp/files/CVE-2023-1999.patch            | 55 +++++++++++++++++++
 meta/recipes-multimedia/webp/libwebp_1.1.0.bb |  4 ++
 meta/recipes-support/vim/vim.inc              |  4 +-
 7 files changed, 77 insertions(+), 7 deletions(-)
 create mode 100644 meta/recipes-multimedia/webp/files/CVE-2023-1999.patch

-- 
2.34.1

[OE-core][dunfell 1/5] libwebp: Fix CVE-2023-1999

[Steve Sakoman]

From: Nikhil R <nikhilar2410@gmail.com>

Add patch to fix CVE-2023-1999

Link: https://github.com/webmproject/libwebp/commit/a486d800b60d0af4cc0836bf7ed8f21e12974129

Signed-off-by: Nikhil R <nikhil.r@kpit.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 .../webp/files/CVE-2023-1999.patch            | 55 +++++++++++++++++++
 meta/recipes-multimedia/webp/libwebp_1.1.0.bb |  4 ++
 2 files changed, 59 insertions(+)
 create mode 100644 meta/recipes-multimedia/webp/files/CVE-2023-1999.patch

diff --git a/meta/recipes-multimedia/webp/files/CVE-2023-1999.patch b/meta/recipes-multimedia/webp/files/CVE-2023-1999.patch
new file mode 100644
index 0000000000..d293ab93ab
--- /dev/null
+++ b/meta/recipes-multimedia/webp/files/CVE-2023-1999.patch
@@ -0,0 +1,55 @@
+From a486d800b60d0af4cc0836bf7ed8f21e12974129 Mon Sep 17 00:00:00 2001
+From: James Zern <jzern@google.com>
+Date: Wed, 22 Feb 2023 22:15:47 -0800
+Subject: [PATCH] EncodeAlphaInternal: clear result->bw on error
+
+This avoids a double free should the function fail prior to
+VP8BitWriterInit() and a previous trial result's buffer carried over.
+Previously in ApplyFiltersAndEncode() trial.bw (with a previous
+iteration's buffer) would be freed, followed by best.bw pointing to the
+same buffer.
+
+Since:
+187d379d add a fallback to ALPHA_NO_COMPRESSION
+
+In addition, check the return value of VP8BitWriterInit() in this
+function.
+
+Bug: webp:603
+Change-Id: Ic258381ee26c8c16bc211d157c8153831c8c6910
+
+CVE: CVE-2023-1999
+Upstream-Status: Backport [https://github.com/webmproject/libwebp/commit/a486d800b60d0af4cc0836bf7ed8f21e12974129]
+Signed-off-by: Nikhil R <nikhil.r@kpit.com>
+---
+ src/enc/alpha_enc.c | 4 +++-
+ 1 file changed, 3 insertions(+), 1 deletion(-)
+
+diff --git a/src/enc/alpha_enc.c b/src/enc/alpha_enc.c
+index f7c02690e3..7d205586fe 100644
+--- a/src/enc/alpha_enc.c
++++ b/src/enc/alpha_enc.c
+@@ -13,6 +13,7 @@
+ 
+ #include <assert.h>
+ #include <stdlib.h>
++#include <string.h>
+ 
+ #include "src/enc/vp8i_enc.h"
+ #include "src/dsp/dsp.h"
+@@ -148,6 +149,7 @@ static int EncodeAlphaInternal(const uint8_t* const data, int width, int height,
+       }
+     } else {
+       VP8LBitWriterWipeOut(&tmp_bw);
++      memset(&result->bw, 0, sizeof(result->bw));
+       return 0;
+     }
+   }
+@@ -162,7 +164,7 @@ static int EncodeAlphaInternal(const uint8_t* const data, int width, int height,
+   header = method | (filter << 2);
+   if (reduce_levels) header |= ALPHA_PREPROCESSED_LEVELS << 4;
+ 
+-  VP8BitWriterInit(&result->bw, ALPHA_HEADER_LEN + output_size);
++  if (!VP8BitWriterInit(&result->bw, ALPHA_HEADER_LEN + output_size)) ok = 0;
+   ok = ok && VP8BitWriterAppend(&result->bw, &header, ALPHA_HEADER_LEN);
+   ok = ok && VP8BitWriterAppend(&result->bw, output, output_size);
diff --git a/meta/recipes-multimedia/webp/libwebp_1.1.0.bb b/meta/recipes-multimedia/webp/libwebp_1.1.0.bb
index 68e5ae2b3c..f449ae750b 100644
--- a/meta/recipes-multimedia/webp/libwebp_1.1.0.bb
+++ b/meta/recipes-multimedia/webp/libwebp_1.1.0.bb
@@ -19,6 +19,10 @@ SRC_URI[sha256sum] = "98a052268cc4d5ece27f76572a7f50293f439c17a98e67c4ea0c7ed6f5
 
 UPSTREAM_CHECK_URI = "http://downloads.webmproject.org/releases/webp/index.html"
 
+SRC_URI += " \
+    file://CVE-2023-1999.patch \
+"
+
 EXTRA_OECONF = " \
     --disable-wic \
     --enable-libwebpmux \
-- 
2.34.1

[OE-core][dunfell 2/5] vim: upgrade 9.0.1429 -> 9.0.1527

[Steve Sakoman]

From: Randy MacLeod <Randy.MacLeod@windriver.com>

Fixes: https://nvd.nist.gov/vuln/detail/CVE-2023-2426
caf642c25 patch 9.0.1499: using uninitialized memory with fuzzy matching

Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 4f9a8df5aca99d0a5c2d2346b27ce7be08e7896c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-support/vim/vim.inc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-support/vim/vim.inc b/meta/recipes-support/vim/vim.inc
index 94eabfa197..800ee40f92 100644
--- a/meta/recipes-support/vim/vim.inc
+++ b/meta/recipes-support/vim/vim.inc
@@ -19,8 +19,8 @@ SRC_URI = "git://github.com/vim/vim.git;branch=master;protocol=https \
            file://no-path-adjust.patch \
            "
 
-PV .= ".1429"
-SRCREV = "1a08a3e2a584889f19b84a27672134649b73da58"
+PV .= ".1527"
+SRCREV = "c28e7a2b2f23dbd246a1ad7ad7aaa6f7ab2e5887"
 
 # Remove when 8.3 is out
 UPSTREAM_VERSION_UNKNOWN = "1"
-- 
2.34.1

[OE-core][dunfell 3/5] systemd-systemctl: fix instance template WantedBy symlink construction

[Steve Sakoman]

From: Martin Siegumfeldt <mns@gomspace.com>

Fix issue of the below instance template systemd service dependency

[Install]
WantedBy=svc-wants@%i.service

creating the symlink (instance "a" example)

/etc/systemd/system/svc-wants@%i.service.wants/svc-wanted-by@a.service

which should be

/etc/systemd/system/svc-wants@a.service.wants/svc-wanted-by@a.service

as implemented by this change.

The functionality appears regressed just after "thud" baseline when the
logic was refactored from shell script into python (commit
925e30cb104ece7bfa48b78144e758a46dc9ec3f)

(From OE-Core rev: 308397f0bb3d6f3d4e9ec2c6a10823184049c9b5)

Signed-off-by: Martin Siegumfeldt <mns@gomspace.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 372b29c8ad270d4d430c26a4e614976c7029afaf)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 .../systemd/systemd-systemctl/systemctl             | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl b/meta/recipes-core/systemd/systemd-systemctl/systemctl
index 6aa2e20465..577c373181 100755
--- a/meta/recipes-core/systemd/systemd-systemctl/systemctl
+++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl
@@ -182,12 +182,19 @@ class SystemdUnit():
 
         raise SystemdUnitNotFoundError(self.root, unit)
 
-    def _process_deps(self, config, service, location, prop, dirstem):
+    def _process_deps(self, config, service, location, prop, dirstem, instance):
         systemdir = self.root / SYSCONFDIR / "systemd" / "system"
 
         target = ROOT / location.relative_to(self.root)
         try:
             for dependent in config.get('Install', prop):
+                # determine whether or not dependent is a template with an actual
+                # instance (i.e. a '@%i')
+                dependent_is_template = re.match(r"[^@]+@(?P<instance>[^\.]*)\.", dependent)
+                if dependent_is_template:
+                    # if so, replace with the actual instance to achieve
+                    # svc-wants@a.service.wants/svc-wanted-by@a.service
+                    dependent = re.sub(dependent_is_template.group('instance'), instance, dependent, 1)
                 wants = systemdir / "{}.{}".format(dependent, dirstem) / service
                 add_link(wants, target)
 
@@ -227,8 +234,8 @@ class SystemdUnit():
         else:
             service = self.unit
 
-        self._process_deps(config, service, path, 'WantedBy', 'wants')
-        self._process_deps(config, service, path, 'RequiredBy', 'requires')
+        self._process_deps(config, service, path, 'WantedBy', 'wants', instance)
+        self._process_deps(config, service, path, 'RequiredBy', 'requires', instance)
 
         try:
             for also in config.get('Install', 'Also'):
-- 
2.34.1

RE: [OE-core][dunfell 3/5] systemd-systemctl: fix instance template WantedBy symlink construction

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Steve Sakoman
> Sent: den 14 juni 2023 16:05
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core][dunfell 3/5] systemd-systemctl: fix instance template
> WantedBy symlink construction
> 
> From: Martin Siegumfeldt <mns@gomspace.com>
> 
> Fix issue of the below instance template systemd service dependency
> 
> [Install]
> WantedBy=svc-wants@%i.service
> 
> creating the symlink (instance "a" example)
> 
> /etc/systemd/system/svc-wants@%i.service.wants/svc-wanted-by@a.service
> 
> which should be
> 
> /etc/systemd/system/svc-wants@a.service.wants/svc-wanted-by@a.service
> 
> as implemented by this change.
> 
> The functionality appears regressed just after "thud" baseline when the
> logic was refactored from shell script into python (commit
> 925e30cb104ece7bfa48b78144e758a46dc9ec3f)
> 
> (From OE-Core rev: 308397f0bb3d6f3d4e9ec2c6a10823184049c9b5)
> 
> Signed-off-by: Martin Siegumfeldt <mns@gomspace.com>
> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> Signed-off-by: Steve Sakoman <steve@sakoman.com>
> (cherry picked from commit 372b29c8ad270d4d430c26a4e614976c7029afaf)
> Signed-off-by: Steve Sakoman <steve@sakoman.com>
> ---
>  .../systemd/systemd-systemctl/systemctl             | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl
> b/meta/recipes-core/systemd/systemd-systemctl/systemctl
> index 6aa2e20465..577c373181 100755
> --- a/meta/recipes-core/systemd/systemd-systemctl/systemctl
> +++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl
> @@ -182,12 +182,19 @@ class SystemdUnit():
> 
>          raise SystemdUnitNotFoundError(self.root, unit)
> 
> -    def _process_deps(self, config, service, location, prop, dirstem):
> +    def _process_deps(self, config, service, location, prop, dirstem, instance):
>          systemdir = self.root / SYSCONFDIR / "systemd" / "system"
> 
>          target = ROOT / location.relative_to(self.root)
>          try:
>              for dependent in config.get('Install', prop):
> +                # determine whether or not dependent is a template with an actual
> +                # instance (i.e. a '@%i')
> +                dependent_is_template = re.match(r"[^@]+@(?P<instance>[^\.]*)\.", dependent)
> +                if dependent_is_template:
> +                    # if so, replace with the actual instance to achieve
> +                    # svc-wants@a.service.wants/svc-wanted-by@a.service
> +                    dependent = re.sub(dependent_is_template.group('instance'), instance, dependent, 1)
>                  wants = systemdir / "{}.{}".format(dependent, dirstem) / service
>                  add_link(wants, target)
> 
> @@ -227,8 +234,8 @@ class SystemdUnit():
>          else:
>              service = self.unit
> 
> -        self._process_deps(config, service, path, 'WantedBy', 'wants')
> -        self._process_deps(config, service, path, 'RequiredBy', 'requires')
> +        self._process_deps(config, service, path, 'WantedBy', 'wants', instance)
> +        self._process_deps(config, service, path, 'RequiredBy', 'requires', instance)
> 
>          try:
>              for also in config.get('Install', 'Also'):
> --
> 2.34.1

You might want to be careful with this one. It broke systemctl for us when it 
hit Mickledore and we have had to backport the version from Langdale for now.

The problem is that it does not handle non-instanced units depending on 
an instanced unit. In our case we have a unit foo.service that contains 
a RequiredBy=bar@0.service, which causes an error like this:

Traceback (most recent call last):
  File ".../recipe-sysroot-native/usr/bin/systemctl", line 366, in <module>
    main()
  File ".../recipe-sysroot-native/usr/bin/systemctl", line 352, in main
    SystemdUnit(root, service).enable()
  File ".../recipe-sysroot-native/usr/bin/systemctl", line 250, in enable
    self._process_deps(config, service, path, 'WantedBy', 'wants', instance)
  File ".../recipe-sysroot-native/usr/bin/systemctl", line 210, in _process_deps
    dependent = re.sub(dependent_is_template.group('instance'), instance, dependent, 1)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py", line 185, in sub
    return _compile(pattern, flags).sub(repl, string, count)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py", line 317, in _subx
    template = _compile_repl(template, pattern)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py", line 308, in _compile_repl
    return _parser.parse_template(repl, pattern)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../recipe-sysroot-native/usr/lib/python3.11/re/_parser.py", line 1000, in parse_template
    s = Tokenizer(source)
        ^^^^^^^^^^^^^^^^^
  File ".../recipe-sysroot-native/usr/lib/python3.11/re/_parser.py", line 226, in __init__
    string = str(string, 'latin1')
             ^^^^^^^^^^^^^^^^^^^^^
TypeError: decoding to str: need a bytes-like object, NoneType found

There are probably other cases too when different kinds of units are 
involved (at least as far as I could gather from the systemd manual).

We have an internal issue to fix this, but I am going on vacation on 
Monday, so I will unfortunately not have any time to look at it until 
after summer.

//Peter

Re: [OE-core][dunfell 3/5] systemd-systemctl: fix instance template WantedBy symlink construction

[Richard Purdie]

On Thu, 2023-06-15 at 20:40 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Steve Sakoman
> > Sent: den 14 juni 2023 16:05
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core][dunfell 3/5] systemd-systemctl: fix instance
> > template
> > WantedBy symlink construction
> > 
> > From: Martin Siegumfeldt <mns@gomspace.com>
> > 
> > Fix issue of the below instance template systemd service dependency
> > 
> > [Install]
> > WantedBy=svc-wants@%i.service
> > 
> > creating the symlink (instance "a" example)
> > 
> > /etc/systemd/system/svc-
> > wants@%i.service.wants/svc-wanted-by@a.service
> > 
> > which should be
> > 
> > /etc/systemd/system/svc-wants@a.service.wants
> > /svc-wanted-by@a.service
> > 
> > as implemented by this change.
> > 
> > The functionality appears regressed just after "thud" baseline when
> > the
> > logic was refactored from shell script into python (commit
> > 925e30cb104ece7bfa48b78144e758a46dc9ec3f)
> > 
> > (From OE-Core rev: 308397f0bb3d6f3d4e9ec2c6a10823184049c9b5)
> > 
> > Signed-off-by: Martin Siegumfeldt <mns@gomspace.com>
> > Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Signed-off-by: Steve Sakoman <steve@sakoman.com>
> > (cherry picked from commit
> > 372b29c8ad270d4d430c26a4e614976c7029afaf)
> > Signed-off-by: Steve Sakoman <steve@sakoman.com>
> > ---
> >  .../systemd/systemd-systemctl/systemctl             | 13
> > ++++++++++---
> >  1 file changed, 10 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > b/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > index 6aa2e20465..577c373181 100755
> > --- a/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > +++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > @@ -182,12 +182,19 @@ class SystemdUnit():
> > 
> >          raise SystemdUnitNotFoundError(self.root, unit)
> > 
> > -    def _process_deps(self, config, service, location, prop,
> > dirstem):
> > +    def _process_deps(self, config, service, location, prop,
> > dirstem, instance):
> >          systemdir = self.root / SYSCONFDIR / "systemd" / "system"
> > 
> >          target = ROOT / location.relative_to(self.root)
> >          try:
> >              for dependent in config.get('Install', prop):
> > +                # determine whether or not dependent is a template
> > with an actual
> > +                # instance (i.e. a '@%i')
> > +                dependent_is_template =
> > re.match(r"[^@]+@(?P<instance>[^\.]*)\.", dependent)
> > +                if dependent_is_template:
> > +                    # if so, replace with the actual instance to
> > achieve
> > +                    #
> > svc-wants@a.service.wants/svc-wanted-by@a.service
> > +                    dependent =
> > re.sub(dependent_is_template.group('instance'), instance,
> > dependent, 1)
> >                  wants = systemdir / "{}.{}".format(dependent,
> > dirstem) / service
> >                  add_link(wants, target)
> > 
> > @@ -227,8 +234,8 @@ class SystemdUnit():
> >          else:
> >              service = self.unit
> > 
> > -        self._process_deps(config, service, path, 'WantedBy',
> > 'wants')
> > -        self._process_deps(config, service, path, 'RequiredBy',
> > 'requires')
> > +        self._process_deps(config, service, path, 'WantedBy',
> > 'wants', instance)
> > +        self._process_deps(config, service, path, 'RequiredBy',
> > 'requires', instance)
> > 
> >          try:
> >              for also in config.get('Install', 'Also'):
> > --
> > 2.34.1
> 
> You might want to be careful with this one. It broke systemctl for us
> when it 
> hit Mickledore and we have had to backport the version from Langdale
> for now.
> 
> The problem is that it does not handle non-instanced units depending
> on 
> an instanced unit. In our case we have a unit foo.service that
> contains 
> a RequiredBy=bar@0.service, which causes an error like this:
> 
> Traceback (most recent call last):
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 366, in
> <module>
>     main()
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 352, in
> main
>     SystemdUnit(root, service).enable()
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 250, in
> enable
>     self._process_deps(config, service, path, 'WantedBy', 'wants',
> instance)
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 210, in
> _process_deps
>     dependent = re.sub(dependent_is_template.group('instance'),
> instance, dependent, 1)
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> ^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py",
> line 185, in sub
>     return _compile(pattern, flags).sub(repl, string, count)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py",
> line 317, in _subx
>     template = _compile_repl(template, pattern)
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py",
> line 308, in _compile_repl
>     return _parser.parse_template(repl, pattern)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/_parser.py",
> line 1000, in parse_template
>     s = Tokenizer(source)
>         ^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/_parser.py",
> line 226, in __init__
>     string = str(string, 'latin1')
>              ^^^^^^^^^^^^^^^^^^^^^
> TypeError: decoding to str: need a bytes-like object, NoneType found
> 
> There are probably other cases too when different kinds of units are 
> involved (at least as far as I could gather from the systemd manual).
> 
> We have an internal issue to fix this, but I am going on vacation on 
> Monday, so I will unfortunately not have any time to look at it until
> after summer.

There is a patch in master recently which may help?

Cheers,

Richard

Re: [OE-core][dunfell 3/5] systemd-systemctl: fix instance template WantedBy symlink construction

[Steve Sakoman]

On Thu, Jun 15, 2023 at 10:41 AM Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Steve Sakoman
> > Sent: den 14 juni 2023 16:05
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core][dunfell 3/5] systemd-systemctl: fix instance template
> > WantedBy symlink construction
> >
> > From: Martin Siegumfeldt <mns@gomspace.com>
> >
> > Fix issue of the below instance template systemd service dependency
> >
> > [Install]
> > WantedBy=svc-wants@%i.service
> >
> > creating the symlink (instance "a" example)
> >
> > /etc/systemd/system/svc-wants@%i.service.wants/svc-wanted-by@a.service
> >
> > which should be
> >
> > /etc/systemd/system/svc-wants@a.service.wants/svc-wanted-by@a.service
> >
> > as implemented by this change.
> >
> > The functionality appears regressed just after "thud" baseline when the
> > logic was refactored from shell script into python (commit
> > 925e30cb104ece7bfa48b78144e758a46dc9ec3f)
> >
> > (From OE-Core rev: 308397f0bb3d6f3d4e9ec2c6a10823184049c9b5)
> >
> > Signed-off-by: Martin Siegumfeldt <mns@gomspace.com>
> > Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Signed-off-by: Steve Sakoman <steve@sakoman.com>
> > (cherry picked from commit 372b29c8ad270d4d430c26a4e614976c7029afaf)
> > Signed-off-by: Steve Sakoman <steve@sakoman.com>
> > ---
> >  .../systemd/systemd-systemctl/systemctl             | 13 ++++++++++---
> >  1 file changed, 10 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > b/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > index 6aa2e20465..577c373181 100755
> > --- a/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > +++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl
> > @@ -182,12 +182,19 @@ class SystemdUnit():
> >
> >          raise SystemdUnitNotFoundError(self.root, unit)
> >
> > -    def _process_deps(self, config, service, location, prop, dirstem):
> > +    def _process_deps(self, config, service, location, prop, dirstem, instance):
> >          systemdir = self.root / SYSCONFDIR / "systemd" / "system"
> >
> >          target = ROOT / location.relative_to(self.root)
> >          try:
> >              for dependent in config.get('Install', prop):
> > +                # determine whether or not dependent is a template with an actual
> > +                # instance (i.e. a '@%i')
> > +                dependent_is_template = re.match(r"[^@]+@(?P<instance>[^\.]*)\.", dependent)
> > +                if dependent_is_template:
> > +                    # if so, replace with the actual instance to achieve
> > +                    # svc-wants@a.service.wants/svc-wanted-by@a.service
> > +                    dependent = re.sub(dependent_is_template.group('instance'), instance, dependent, 1)
> >                  wants = systemdir / "{}.{}".format(dependent, dirstem) / service
> >                  add_link(wants, target)
> >
> > @@ -227,8 +234,8 @@ class SystemdUnit():
> >          else:
> >              service = self.unit
> >
> > -        self._process_deps(config, service, path, 'WantedBy', 'wants')
> > -        self._process_deps(config, service, path, 'RequiredBy', 'requires')
> > +        self._process_deps(config, service, path, 'WantedBy', 'wants', instance)
> > +        self._process_deps(config, service, path, 'RequiredBy', 'requires', instance)
> >
> >          try:
> >              for also in config.get('Install', 'Also'):
> > --
> > 2.34.1
>
> You might want to be careful with this one. It broke systemctl for us when it
> hit Mickledore and we have had to backport the version from Langdale for now.

Thanks for reviewing.  I'll drop this from dunfell for now and will
look for the potential fix in master for mickledore.

Steve

> The problem is that it does not handle non-instanced units depending on
> an instanced unit. In our case we have a unit foo.service that contains
> a RequiredBy=bar@0.service, which causes an error like this:
>
> Traceback (most recent call last):
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 366, in <module>
>     main()
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 352, in main
>     SystemdUnit(root, service).enable()
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 250, in enable
>     self._process_deps(config, service, path, 'WantedBy', 'wants', instance)
>   File ".../recipe-sysroot-native/usr/bin/systemctl", line 210, in _process_deps
>     dependent = re.sub(dependent_is_template.group('instance'), instance, dependent, 1)
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py", line 185, in sub
>     return _compile(pattern, flags).sub(repl, string, count)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py", line 317, in _subx
>     template = _compile_repl(template, pattern)
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/__init__.py", line 308, in _compile_repl
>     return _parser.parse_template(repl, pattern)
>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/_parser.py", line 1000, in parse_template
>     s = Tokenizer(source)
>         ^^^^^^^^^^^^^^^^^
>   File ".../recipe-sysroot-native/usr/lib/python3.11/re/_parser.py", line 226, in __init__
>     string = str(string, 'latin1')
>              ^^^^^^^^^^^^^^^^^^^^^
> TypeError: decoding to str: need a bytes-like object, NoneType found
>
> There are probably other cases too when different kinds of units are
> involved (at least as far as I could gather from the systemd manual).
>
> We have an internal issue to fix this, but I am going on vacation on
> Monday, so I will unfortunately not have any time to look at it until
> after summer.
>
> //Peter

[OE-core][dunfell 4/5] e2fsprogs: fix ptest bug for second running

[Steve Sakoman]

From: Qiu Tingting <qiutt@fujitsu.com>

At second running, there are four new failed case:
  d_loaddump f_bigalloc_badinode f_bigalloc_orphan_list f_dup_resize

The test_data.tmp is necessary, but it is deleted by run-ptest.
So it should be restored after testing.

Signed-off-by: Qiu Tingting <qiutt@fujitsu.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit d0d08dd9a8a179e25b9cfcbac696c1d212a1910c)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest | 1 +
 meta/recipes-devtools/e2fsprogs/e2fsprogs_1.45.7.bb | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest b/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
index c97c0377e9..279923db8e 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs/run-ptest
@@ -8,3 +8,4 @@ rm -f *.tmp
 rm -f *.ok
 rm -f *.failed
 rm -f *.log
+cp ../data/test_data.tmp ./
diff --git a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.45.7.bb b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.45.7.bb
index b1abf6b858..565c433866 100644
--- a/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.45.7.bb
+++ b/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.45.7.bb
@@ -144,4 +144,7 @@ do_install_ptest() {
 
         install -d ${D}${PTEST_PATH}/lib
         install -m 0644 ${B}/lib/config.h  ${D}${PTEST_PATH}/lib/
+
+        install -d ${D}${PTEST_PATH}/data
+        install -m 0644 ${B}/tests/test_data.tmp ${D}${PTEST_PATH}/data/
 }
-- 
2.34.1

[OE-core][dunfell 5/5] selftest/reproducible: Allow native/cross reuse in test

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We don't compare reproducibility of the native/cross components, only the
target ones. With the long build times of rust-native, the test now takes
crazy lengths of time so this tweak should allow us to reuse native/cross
artefacts from sstate whilst still testing the target output is
reproducible.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b494d83c639a877cefeb7cbab6d37195e492f059)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/lib/oeqa/selftest/cases/reproducible.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index adaabee47b..be4cdcc429 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -188,7 +188,7 @@ class ReproducibleTests(OESelftestTestCase):
 
     def setUpLocal(self):
         super().setUpLocal()
-        needed_vars = ['TOPDIR', 'TARGET_PREFIX', 'BB_NUMBER_THREADS']
+        needed_vars = ['TOPDIR', 'TARGET_PREFIX', 'BB_NUMBER_THREADS', 'BB_HASHSERVE']
         bb_vars = get_bb_vars(needed_vars)
         for v in needed_vars:
             setattr(self, v.lower(), bb_vars[v])
@@ -259,7 +259,7 @@ class ReproducibleTests(OESelftestTestCase):
             # mirror, forcing a complete build from scratch
             config += textwrap.dedent('''\
                 SSTATE_DIR = "${TMPDIR}/sstate"
-                SSTATE_MIRRORS = ""
+                SSTATE_MIRRORS = "file://.*/.*-native.*  http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH file://.*/.*-cross.*  http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
                 ''')
 
         self.logger.info("Building %s (sstate%s allowed)..." % (name, '' if use_sstate else ' NOT'))
-- 
2.34.1