[PATCHv2 1/5] lib/oe/patch: Make extractPatches() not extract ignored commits

[PATCHv2 1/5] lib/oe/patch: Make extractPatches() not extract ignored commits

[Peter Kjellerstedt]

If a commit is marked with "%% ignore" it means it is used by devtool to
keep track of changes to the source code that are not the result of
running do_patch(). These changes need to actually be ignored when
extracting the patches as they typically make no sense as actual patches
in a recipe.

This also adds a new test for oe-selftest that verifies that there are
no patches generated from ignored commits.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---

PATCHv2: Add EXCLUDE_FROM_WORLD = "1" to the sysdig-selftest recipe.

 .../0055-Add-cstdint-for-uintXX_t-types.patch | 38 +++++++++++
 ...099-cmake-Pass-PROBE_NAME-via-CFLAGS.patch | 29 ++++++++
 .../sysdig/sysdig-selftest_0.28.0.bb          | 66 +++++++++++++++++++
 meta/lib/oe/patch.py                          | 16 ++---
 meta/lib/oeqa/selftest/cases/devtool.py       | 46 +++++++++++++
 5 files changed, 187 insertions(+), 8 deletions(-)
 create mode 100644 meta-selftest/recipes-extended/sysdig/sysdig-selftest/0055-Add-cstdint-for-uintXX_t-types.patch
 create mode 100644 meta-selftest/recipes-extended/sysdig/sysdig-selftest/0099-cmake-Pass-PROBE_NAME-via-CFLAGS.patch
 create mode 100644 meta-selftest/recipes-extended/sysdig/sysdig-selftest_0.28.0.bb

diff --git a/meta-selftest/recipes-extended/sysdig/sysdig-selftest/0055-Add-cstdint-for-uintXX_t-types.patch b/meta-selftest/recipes-extended/sysdig/sysdig-selftest/0055-Add-cstdint-for-uintXX_t-types.patch
new file mode 100644
index 0000000000..e564958dad
--- /dev/null
+++ b/meta-selftest/recipes-extended/sysdig/sysdig-selftest/0055-Add-cstdint-for-uintXX_t-types.patch
@@ -0,0 +1,38 @@
+From 3d076ea588eb3c7f334133b4c31172a14beadf5b Mon Sep 17 00:00:00 2001
+From: Khem Raj <raj.khem@gmail.com>
+Date: Thu, 2 Feb 2023 20:18:27 -0800
+Subject: [PATCH] Add <cstdint> for uintXX_t types
+
+gcc 13 moved some includes around and as a result <cstdint> is no
+longer transitively included [1]. Explicitly include it
+for uintXX_t.
+
+[1] https://gcc.gnu.org/gcc-13/porting_to.html#header-dep-changes
+
+Upstream-Status: Submitted [https://github.com/falcosecurity/libs/pull/862]
+Signed-off-by: Khem Raj <raj.khem@gmail.com>
+---
+ userspace/libsinsp/filter/parser.h | 1 +
+ userspace/libsinsp/filter_value.h  | 1 +
+ 2 files changed, 2 insertions(+)
+
+--- a/userspace/libsinsp/filter/parser.h
++++ b/userspace/libsinsp/filter/parser.h
+@@ -18,6 +18,7 @@ limitations under the License.
+ #pragma once
+ 
+ #include "ast.h"
++#include <cstdint>
+ 
+ //
+ // Context-free Grammar for Sinsp Filters
+--- a/userspace/libsinsp/filter_value.h
++++ b/userspace/libsinsp/filter_value.h
+@@ -18,6 +18,7 @@ limitations under the License.
+ #pragma once
+ 
+ #include <string.h>
++#include <cstdint>
+ #include <utility>
+ 
+ // Used for CO_IN/CO_PMATCH filterchecks using PT_CHARBUFs to allow
diff --git a/meta-selftest/recipes-extended/sysdig/sysdig-selftest/0099-cmake-Pass-PROBE_NAME-via-CFLAGS.patch b/meta-selftest/recipes-extended/sysdig/sysdig-selftest/0099-cmake-Pass-PROBE_NAME-via-CFLAGS.patch
new file mode 100644
index 0000000000..903ccdf36a
--- /dev/null
+++ b/meta-selftest/recipes-extended/sysdig/sysdig-selftest/0099-cmake-Pass-PROBE_NAME-via-CFLAGS.patch
@@ -0,0 +1,29 @@
+From ed8969a233adb6bf701de96d0fd0570e5ddcc787 Mon Sep 17 00:00:00 2001
+From: Khem Raj <raj.khem@gmail.com>
+Date: Mon, 21 Mar 2022 19:35:48 -0700
+Subject: [PATCH] cmake: Pass PROBE_NAME via CFLAGS
+
+This helps compliation of driver code where its calling modprobe on the
+given kernel module via system() API
+
+Upstream-Status: Pending
+Signed-off-by: Khem Raj <raj.khem@gmail.com>
+---
+ CMakeLists.txt | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 7dceb7ae..e156c36f 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -149,6 +149,7 @@ if(CMAKE_SYSTEM_NAME MATCHES "Linux")
+ 	if(NOT DEFINED PROBE_NAME)
+ 		set(PROBE_NAME "scap")
+ 	endif()
++	add_definitions(-DPROBE_NAME="${PROBE_NAME}")
+ 
+ 	set(DRIVERS_REPO "https://download.sysdig.com/scap-drivers")
+ 	
+-- 
+2.35.1
+
diff --git a/meta-selftest/recipes-extended/sysdig/sysdig-selftest_0.28.0.bb b/meta-selftest/recipes-extended/sysdig/sysdig-selftest_0.28.0.bb
new file mode 100644
index 0000000000..2ce85fe451
--- /dev/null
+++ b/meta-selftest/recipes-extended/sysdig/sysdig-selftest_0.28.0.bb
@@ -0,0 +1,66 @@
+SUMMARY = "A New System Troubleshooting Tool Built for the Way You Work"
+DESCRIPTION = "Sysdig is open source, system-level exploration: capture \
+system state and activity from a running Linux instance, then save, \
+filter and analyze."
+HOMEPAGE = "http://www.sysdig.org/"
+LICENSE = "Apache-2.0 & (MIT | GPL-2.0-only)"
+LIC_FILES_CHKSUM = "file://COPYING;md5=f8fee3d59797546cffab04f3b88b2d44"
+
+inherit cmake pkgconfig
+
+#OECMAKE_GENERATOR = "Unix Makefiles"
+JIT ?= "jit"
+JIT:mipsarchn32 = ""
+JIT:mipsarchn64 = ""
+JIT:riscv64 = ""
+JIT:riscv32 = ""
+JIT:powerpc = ""
+JIT:powerpc64le = ""
+JIT:powerpc64 = ""
+
+#DEPENDS += "libb64 lua${JIT} zlib c-ares grpc-native grpc curl ncurses jsoncpp \
+#            tbb jq openssl elfutils protobuf protobuf-native jq-native valijson"
+RDEPENDS:${PN} = "bash"
+
+SRC_URI = "git://github.com/draios/sysdig.git;branch=dev;protocol=https;name=sysdig \
+           git://github.com/falcosecurity/libs;protocol=https;branch=master;name=falco;subdir=git/falcosecurity-libs \
+           file://0055-Add-cstdint-for-uintXX_t-types.patch;patchdir=./falcosecurity-libs \
+           file://0099-cmake-Pass-PROBE_NAME-via-CFLAGS.patch \
+           "
+SRCREV_sysdig = "4fb6288275f567f63515df0ff0a6518043ecfa9b"
+SRCREV_falco= "caa0e4d0044fdaaebab086592a97f0c7f32aeaa9"
+
+SRCREV_FORMAT = "sysdig_falco"
+
+S = "${WORKDIR}/git"
+
+EXTRA_OECMAKE = "\
+                -DBUILD_DRIVER=OFF \
+                -DMINIMAL_BUILD=ON \
+                -DUSE_BUNDLED_DEPS=OFF \
+                -DCREATE_TEST_TARGETS=OFF \
+                -DDIR_ETC=${sysconfdir} \
+                -DLUA_INCLUDE_DIR=${STAGING_INCDIR}/luajit-2.1 \
+                -DFALCOSECURITY_LIBS_SOURCE_DIR=${S}/falcosecurity-libs \
+                -DVALIJSON_INCLUDE=${STAGING_INCDIR}/valijson \
+"
+
+#CMAKE_VERBOSE = "VERBOSE=1"
+
+FILES:${PN} += " \
+    ${DIR_ETC}/* \
+    ${datadir}/zsh/* \
+    ${prefix}/src/*  \
+"
+# Use getaddrinfo_a is a GNU extension in libsinsp
+# It should be fixed in sysdig, until then disable
+# on musl
+# Something like this https://code.videolan.org/ePirat/vlc/-/commit/01fd9fe4c7f6c5558f7345f38abf0152e17853ab  is needed to fix it
+COMPATIBLE_HOST:libc-musl = "null"
+COMPATIBLE_HOST:mips = "null"
+COMPATIBLE_HOST:riscv64 = "null"
+COMPATIBLE_HOST:riscv32 = "null"
+COMPATIBLE_HOST:powerpc = "null"
+COMPATIBLE_HOST:powerpc64le = "null"
+
+EXCLUDE_FROM_WORLD = "1"
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index d5ad4f3dc1..70cdb1d9c0 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -474,9 +474,9 @@ class GitApplyTree(PatchTree):
                 out = runcmd(["sh", "-c", " ".join(shellcmd)], os.path.join(tree, name))
                 if out:
                     for srcfile in out.split():
+                        outfile = os.path.basename(srcfile)
                         for encoding in ['utf-8', 'latin-1']:
                             patchlines = []
-                            outfile = None
                             try:
                                 with open(srcfile, 'r', encoding=encoding, newline='') as f:
                                     for line in f:
@@ -484,7 +484,8 @@ class GitApplyTree(PatchTree):
                                             outfile = line.split()[-1].strip()
                                             continue
                                         if line.startswith(GitApplyTree.ignore_commit_prefix):
-                                            continue
+                                            outfile = None
+                                            break
                                         patchlines.append(line)
                             except UnicodeDecodeError:
                                 continue
@@ -492,12 +493,11 @@ class GitApplyTree(PatchTree):
                         else:
                             raise PatchError('Unable to find a character encoding to decode %s' % srcfile)
 
-                        if not outfile:
-                            outfile = os.path.basename(srcfile)
-                        bb.utils.mkdirhier(os.path.join(outdir, name))
-                        with open(os.path.join(outdir, name, outfile), 'w') as of:
-                            for line in patchlines:
-                                of.write(line)
+                        if outfile:
+                            bb.utils.mkdirhier(os.path.join(outdir, name))
+                            with open(os.path.join(outdir, name, outfile), 'w') as of:
+                                for line in patchlines:
+                                    of.write(line)
         finally:
             shutil.rmtree(tempdir)
 
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index d76b974fbb..c4dcdb4550 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -2228,6 +2228,52 @@ class DevtoolUpgradeTests(DevtoolBase):
         if files:
             self.fail('Unexpected file(s) copied next to bbappend: %s' % ', '.join(files))
 
+    def test_devtool_finish_update_patch(self):
+        # This test uses a modified version of the sysdig recipe from meta-oe.
+        # - The patches have been renamed.
+        # - The dependencies are commented out since the recipe is not being
+        #   built.
+        #
+        # The sysdig recipe is interesting in that it fetches two different Git
+        # repositories, and there are patches for both. This leads to that
+        # devtool will create ignore commits as it uses Git submodules to keep
+        # track of the second repository.
+        #
+        # This test will verify that the ignored commits actually are ignored
+        # when a commit in between is modified. It will also verify that the
+        # updated patch keeps its original name.
+
+        # Check preconditions
+        self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        # Try modifying a recipe
+        self.track_for_cleanup(self.workspacedir)
+        recipe = 'sysdig-selftest'
+        recipefile = get_bb_var('FILE', recipe)
+        recipedir = os.path.dirname(recipefile)
+        result = runCmd('git status --porcelain .', cwd=recipedir)
+        if result.output.strip():
+            self.fail('Recipe directory for %s contains uncommitted changes' % recipe)
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool modify %s %s' % (recipe, tempdir))
+        self.add_command_to_tearDown('cd %s; rm %s/*; git checkout %s %s' % (recipedir, recipe, recipe, os.path.basename(recipefile)))
+        self.assertExists(os.path.join(tempdir, 'CMakeLists.txt'), 'Extracted source could not be found')
+        # Make a change to one of the existing commits
+        result = runCmd('echo "# A comment " >> CMakeLists.txt', cwd=tempdir)
+        result = runCmd('git status --porcelain', cwd=tempdir)
+        self.assertIn('M CMakeLists.txt', result.output)
+        result = runCmd('git commit --fixup HEAD^ CMakeLists.txt', cwd=tempdir)
+        result = runCmd('git show -s --format=%s', cwd=tempdir)
+        self.assertIn('fixup! cmake: Pass PROBE_NAME via CFLAGS', result.output)
+        result = runCmd('GIT_SEQUENCE_EDITOR=true git rebase -i --autosquash devtool-base', cwd=tempdir)
+        result = runCmd('devtool finish %s meta-selftest' % recipe)
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output, 'Recipe should have been reset by finish but wasn\'t')
+        self.assertNotExists(os.path.join(self.workspacedir, 'recipes', recipe), 'Recipe directory should not exist after finish')
+        expected_status = [(' M', '.*/0099-cmake-Pass-PROBE_NAME-via-CFLAGS.patch$')]
+        self._check_repo_status(recipedir, expected_status)
+
     def test_devtool_rename(self):
         # Check preconditions
         self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')

[PATCHv2 2/5] lib/oe/patch: Add GitApplyTree.commitIgnored()

[Peter Kjellerstedt]

This function can be used to create a commit that devtool will ignore
when creating/updating the patches.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---

PATCHv2: No changes.

 meta/lib/oe/patch.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 70cdb1d9c0..3ded5f3601 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -460,6 +460,16 @@ class GitApplyTree(PatchTree):
             cmd.append('--date="%s"' % date)
         return (tmpfile, cmd)
 
+    @staticmethod
+    def commitIgnored(subject, dir=None, files=None, d=None):
+        if files:
+            runcmd(['git', 'add'] + files, dir)
+        message = "%s\n\n%s" % (subject, GitApplyTree.ignore_commit_prefix)
+        cmd = ["git"]
+        GitApplyTree.gitCommandUserOptions(cmd, d=d)
+        cmd += ["commit", "-m", message, "--no-verify"]
+        runcmd(cmd, dir)
+
     @staticmethod
     def extractPatches(tree, startcommits, outdir, paths=None):
         import tempfile

[PATCHv2 3/5] devtool: Make use of oe.patch.GitApplyTree.commitIgnored()

[Peter Kjellerstedt]

This makes use of the oe.patch.GitApplyTree.commitIgnored() function to
create commits that shall be ignored by `devtool finish`.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---

PATCHv2: No changes.

 scripts/lib/devtool/__init__.py | 4 +---
 scripts/lib/devtool/standard.py | 6 +-----
 2 files changed, 2 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index c7fc3cfc41..6133c1c5b4 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -253,9 +253,7 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
                     bb.process.run('git submodule add %s %s' % (remote_url, os.path.relpath(root, os.path.join(root, ".."))), cwd=os.path.join(root, ".."))
                     found = True
                 if found:
-                    useroptions = []
-                    oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d)
-                    bb.process.run('git %s commit -m "Adding additionnal submodule from SRC_URI\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=os.path.join(root, ".."))
+                    oe.patch.GitApplyTree.commitIgnored("Add additional submodule from SRC_URI", dir=os.path.join(root, ".."), d=d)
                     found = False
     if os.path.exists(os.path.join(repodir, '.gitmodules')):
         bb.process.run('git submodule foreach --recursive  "git tag -f %s"' % basetag, cwd=repodir)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ccb7ea851b..6d7fd17fbd 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -484,11 +484,7 @@ def symlink_oelocal_files_srctree(rd, srctree):
                     os.symlink('oe-local-files/%s' % fn, destpth)
                 addfiles.append(os.path.join(relpth, fn))
         if addfiles:
-            bb.process.run('git add %s' % ' '.join(addfiles), cwd=srctree)
-            useroptions = []
-            oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=rd)
-            bb.process.run('git %s commit -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree)
-
+            oe.patch.GitApplyTree.commitIgnored("Add local file symlinks", dir=srctree, files=addfiles, d=rd)
 
 def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil, no_overrides=False):
     """Extract sources of a recipe"""

Re: [OE-core] [PATCHv2 3/5] devtool: Make use of oe.patch.GitApplyTree.commitIgnored()

[Ross Burton]

On 19 Feb 2024, at 01:28, Peter Kjellerstedt via lists.openembedded.org <peter.kjellerstedt=axis.com@lists.openembedded.org> wrote:
> 
> This makes use of the oe.patch.GitApplyTree.commitIgnored() function to
> create commits that shall be ignored by `devtool finish`.

If we’re using notes to store the filename info, couldn’t the ignores state also be a note?

Ross

RE: [OE-core] [PATCHv2 3/5] devtool: Make use of oe.patch.GitApplyTree.commitIgnored()

[Peter Kjellerstedt]

> -----Original Message-----
> From: Ross Burton <Ross.Burton@arm.com>
> Sent: den 19 februari 2024 13:54
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCHv2 3/5] devtool: Make use of
> oe.patch.GitApplyTree.commitIgnored()
> 
> On 19 Feb 2024, at 01:28, Peter Kjellerstedt via lists.openembedded.org
> <peter.kjellerstedt=axis.com@lists.openembedded.org> wrote:
> >
> > This makes use of the oe.patch.GitApplyTree.commitIgnored() function to
> > create commits that shall be ignored by `devtool finish`.
> 
> If we’re using notes to store the filename info, couldn’t the ignores
> state also be a note?
> 
> Ross

Yes, that comes with patch 5. This was just a preparation to get the three 
different places that create ignore commits gathered into one place.

//Peter

[PATCHv2 4/5] patch.bbclass: Make use of oe.patch.GitApplyTree.commitIgnored()

[Peter Kjellerstedt]

This makes use of the oe.patch.GitApplyTree.commitIgnored() function to
create commits that shall be ignored by `devtool finish`.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---

PATCHv2: No changes.

 meta/classes-global/patch.bbclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/meta/classes-global/patch.bbclass b/meta/classes-global/patch.bbclass
index e3157c7b18..e5786b1c9a 100644
--- a/meta/classes-global/patch.bbclass
+++ b/meta/classes-global/patch.bbclass
@@ -79,9 +79,7 @@ python patch_task_postfunc() {
                         bb.process.run('git checkout patches', cwd=srcsubdir)
         stdout, _ = bb.process.run('git status --porcelain .', cwd=srcsubdir)
         if stdout:
-            useroptions = []
-            oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d)
-            bb.process.run('git add .; git %s commit -a -m "Committing changes from %s\n\n%s"' % (' '.join(useroptions), func, oe.patch.GitApplyTree.ignore_commit_prefix + ' - from %s' % func), cwd=srcsubdir)
+            oe.patch.GitApplyTree.commitIgnored("Add changes from %s" % func, dir=srcsubdir, files=['.'], d=d)
 }
 
 def src_patches(d, all=False, expand=True):

Patchtest results for [PATCHv2 4/5] patch.bbclass: Make use of oe.patch.GitApplyTree.commitIgnored()

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3139 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/PATCHv2-4-5-patch.bbclass-Make-use-of-oe.patch.GitApplyTree.commitIgnored.patch

FAIL: test max line length: Patch line too long (current length 208, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)

SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

[PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches

[Peter Kjellerstedt]

The old way of keeping track of the filenames for the patches that
correspond to the commits was to add a special comment line to the end
of the commit message, e.g., "%% original patch: <filename>", using a
temporary git hook. This method had some drawbacks, e.g.:

* It caused problems if one wanted to push the commits upstream as the
  comment line had to be manually removed.
* The comment line would end up in patches if someone used git
  format-path rather than devtool finish to generate the patches.
* The comment line could interfere with global Git hooks used to
  validate the format of the Git commit message.
* When regenerating patches with `devtool finish --force-patch-refresh`,
  the process typically resulted in adding empty lines to the end of the
  commit messages in the updated patches.

A better way of keeping track of the patch filenames is to use Git
notes. This way the commit messages remain unaffected, but the
information is still shown when, e.g., doing `git log`. A special Git
notes space, refs/notes/devtool, is used to not intefere with the
default Git notes. It is configured to be shown in, e.g., `git log` and
to survive rewrites (i.e., `git commit --amend` and `git rebase`).

Since there is no longer any need for a temporary Git hook, the code
that manipulated the .git/hooks directory has also been removed. To
avoid potential problems due to global Git hooks, --no-verify was added
to the `git commit` command.

To not cause troubles for those who have done `devtool modify` for a
recipe with the old solution and then do `devtool finish` with the new
solution, the code will fall back to look for the old strings in the
commit message if no Git note can be found.

While not technically motivated like above, the way to keep track of
ignored commits is also changed to use Git notes to avoid having
different methods to store similar information.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---

PATCHv2:
* Remove the --fixed-value option from calls to git config. It was
  added in Git version 2.30, but OE Core currently only requires Git
  version 1.8.3.1.
* Work around a bug (or feature?) in git rebase related to commits with
  notes that are completely absorbed by already existing commits during
  a rebase.
* Configure notes.rewriteMode = ignore. This is part of the solution to
  the above problem.

 meta/lib/oe/patch.py                    | 109 +++++++++++++++---------
 meta/lib/oeqa/selftest/cases/devtool.py |   9 +-
 scripts/lib/devtool/standard.py         |  15 ++--
 scripts/lib/devtool/upgrade.py          |  24 +++++-
 4 files changed, 103 insertions(+), 54 deletions(-)

diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 3ded5f3601..60a0cc8291 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -294,8 +294,9 @@ class PatchTree(PatchSet):
         self.Pop(all=True)
 
 class GitApplyTree(PatchTree):
-    patch_line_prefix = '%% original patch'
-    ignore_commit_prefix = '%% ignore'
+    notes_ref = "refs/notes/devtool"
+    original_patch = 'original patch'
+    ignore_commit = 'ignore'
 
     def __init__(self, dir, d):
         PatchTree.__init__(self, dir, d)
@@ -452,7 +453,7 @@ class GitApplyTree(PatchTree):
         # Prepare git command
         cmd = ["git"]
         GitApplyTree.gitCommandUserOptions(cmd, commituser, commitemail)
-        cmd += ["commit", "-F", tmpfile]
+        cmd += ["commit", "-F", tmpfile, "--no-verify"]
         # git doesn't like plain email addresses as authors
         if author and '<' in author:
             cmd.append('--author="%s"' % author)
@@ -460,15 +461,53 @@ class GitApplyTree(PatchTree):
             cmd.append('--date="%s"' % date)
         return (tmpfile, cmd)
 
+    @staticmethod
+    def addNote(repo, ref, key, value=None):
+        note = key + (": %s" % value if value else "")
+        notes_ref = GitApplyTree.notes_ref
+        runcmd(["git", "config", "notes.rewriteMode", "ignore"], repo)
+        runcmd(["git", "config", "notes.displayRef", notes_ref, notes_ref], repo)
+        runcmd(["git", "config", "notes.rewriteRef", notes_ref, notes_ref], repo)
+        runcmd(["git", "notes", "--ref", notes_ref, "append", "-m", note, ref], repo)
+
+    @staticmethod
+    def removeNote(repo, ref, key):
+        notes = GitApplyTree.getNotes(repo, ref)
+        notes = {k: v for k, v in notes.items() if k != key and not k.startswith(key + ":")}
+        runcmd(["git", "notes", "--ref", GitApplyTree.notes_ref, "remove", "--ignore-missing", ref], repo)
+        for note, value in notes.items():
+            GitApplyTree.addNote(repo, ref, note, value)
+
+    @staticmethod
+    def getNotes(repo, ref):
+        import re
+
+        note = None
+        try:
+            note = runcmd(["git", "notes", "--ref", GitApplyTree.notes_ref, "show", ref], repo)
+            prefix = ""
+        except CmdError:
+            note = runcmd(['git', 'show', '-s', '--format=%B', ref], repo)
+            prefix = "%% "
+
+        note_re = re.compile(r'^%s(.*?)(?::\s*(.*))?$' % prefix)
+        notes = dict()
+        for line in note.splitlines():
+            m = note_re.match(line)
+            if m:
+                notes[m.group(1)] = m.group(2)
+
+        return notes
+
     @staticmethod
     def commitIgnored(subject, dir=None, files=None, d=None):
         if files:
             runcmd(['git', 'add'] + files, dir)
-        message = "%s\n\n%s" % (subject, GitApplyTree.ignore_commit_prefix)
         cmd = ["git"]
         GitApplyTree.gitCommandUserOptions(cmd, d=d)
-        cmd += ["commit", "-m", message, "--no-verify"]
+        cmd += ["commit", "-m", subject, "--no-verify"]
         runcmd(cmd, dir)
+        GitApplyTree.addNote(dir, "HEAD", GitApplyTree.ignore_commit)
 
     @staticmethod
     def extractPatches(tree, startcommits, outdir, paths=None):
@@ -484,18 +523,20 @@ class GitApplyTree(PatchTree):
                 out = runcmd(["sh", "-c", " ".join(shellcmd)], os.path.join(tree, name))
                 if out:
                     for srcfile in out.split():
-                        outfile = os.path.basename(srcfile)
+                        # This loop, which is used to remove any line that
+                        # starts with "%% original patch", is kept for backwards
+                        # compatibility. If/when that compatibility is dropped,
+                        # it can be replaced with code to just read the first
+                        # line of the patch file to get the SHA-1, and the code
+                        # below that writes the modified patch file can be
+                        # replaced with a simple file move.
                         for encoding in ['utf-8', 'latin-1']:
                             patchlines = []
                             try:
                                 with open(srcfile, 'r', encoding=encoding, newline='') as f:
                                     for line in f:
-                                        if line.startswith(GitApplyTree.patch_line_prefix):
-                                            outfile = line.split()[-1].strip()
+                                        if line.startswith("%% " + GitApplyTree.original_patch):
                                             continue
-                                        if line.startswith(GitApplyTree.ignore_commit_prefix):
-                                            outfile = None
-                                            break
                                         patchlines.append(line)
                             except UnicodeDecodeError:
                                 continue
@@ -503,11 +544,16 @@ class GitApplyTree(PatchTree):
                         else:
                             raise PatchError('Unable to find a character encoding to decode %s' % srcfile)
 
-                        if outfile:
-                            bb.utils.mkdirhier(os.path.join(outdir, name))
-                            with open(os.path.join(outdir, name, outfile), 'w') as of:
-                                for line in patchlines:
-                                    of.write(line)
+                        sha1 = patchlines[0].split()[1]
+                        notes = GitApplyTree.getNotes(os.path.join(tree, name), sha1)
+                        if GitApplyTree.ignore_commit in notes:
+                            continue
+                        outfile = notes.get(GitApplyTree.original_patch, os.path.basename(srcfile))
+
+                        bb.utils.mkdirhier(os.path.join(outdir, name))
+                        with open(os.path.join(outdir, name, outfile), 'w') as of:
+                            for line in patchlines:
+                                of.write(line)
         finally:
             shutil.rmtree(tempdir)
 
@@ -555,28 +601,11 @@ class GitApplyTree(PatchTree):
 
             return runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
 
-        # Add hooks which add a pointer to the original patch file name in the commit message
         reporoot = (runcmd("git rev-parse --show-toplevel".split(), self.dir) or '').strip()
         if not reporoot:
             raise Exception("Cannot get repository root for directory %s" % self.dir)
-        gitdir = (runcmd("git rev-parse --absolute-git-dir".split(), self.dir) or '').strip()
-        if not gitdir:
-            raise Exception("Cannot get gitdir for directory %s" % self.dir)
-        hooks_dir = os.path.join(gitdir, 'hooks')
-        hooks_dir_backup = hooks_dir + '.devtool-orig'
-        if os.path.lexists(hooks_dir_backup):
-            raise Exception("Git hooks backup directory already exists: %s" % hooks_dir_backup)
-        if os.path.lexists(hooks_dir):
-            shutil.move(hooks_dir, hooks_dir_backup)
-        os.mkdir(hooks_dir)
-        commithook = os.path.join(hooks_dir, 'commit-msg')
-        applyhook = os.path.join(hooks_dir, 'applypatch-msg')
-        with open(commithook, 'w') as f:
-            # NOTE: the formatting here is significant; if you change it you'll also need to
-            # change other places which read it back
-            f.write('echo "\n%s: $PATCHFILE" >> $1' % GitApplyTree.patch_line_prefix)
-        os.chmod(commithook, 0o755)
-        shutil.copy2(commithook, applyhook)
+
+        patch_applied = True
         try:
             patchfilevar = 'PATCHFILE="%s"' % os.path.basename(patch['file'])
             if self._need_dirty_check():
@@ -587,7 +616,7 @@ class GitApplyTree(PatchTree):
                     pass
                 else:
                     if output:
-                        # The tree is dirty, not need to try to apply patches with git anymore
+                        # The tree is dirty, no need to try to apply patches with git anymore
                         # since they fail, fallback directly to patch
                         output = PatchTree._applypatch(self, patch, force, reverse, run)
                         output += self._commitpatch(patch, patchfilevar)
@@ -620,10 +649,12 @@ class GitApplyTree(PatchTree):
                     output = PatchTree._applypatch(self, patch, force, reverse, run)
                 output += self._commitpatch(patch, patchfilevar)
                 return output
+        except:
+            patch_applied = False
+            raise
         finally:
-            shutil.rmtree(hooks_dir)
-            if os.path.lexists(hooks_dir_backup):
-                shutil.move(hooks_dir_backup, hooks_dir)
+            if patch_applied:
+                GitApplyTree.addNote(self.dir, "HEAD", GitApplyTree.original_patch, os.path.basename(patch['file']))
 
 
 class QuiltTree(PatchSet):
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index c4dcdb4550..d37848bdef 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -989,9 +989,10 @@ class DevtoolModifyTests(DevtoolBase):
         self.assertIn(tempdir, result.output)
         # Check git repo
         self._check_src_repo(tempdir)
-        # Check that the patch is correctly applied
-        # last commit message in the tree must contain
-        # %% original patch: <patchname>
+        # Check that the patch is correctly applied.
+        # The last commit message in the tree must contain the following note:
+        # Notes (devtool):
+        #     original patch: <patchname>
         # ..
         patchname = None
         for uri in src_uri:
@@ -999,7 +1000,7 @@ class DevtoolModifyTests(DevtoolBase):
                 patchname = uri.replace("file://", "").partition('.patch')[0] + '.patch'
         self.assertIsNotNone(patchname)
         result = runCmd('git -C %s log -1' % tempdir)
-        self.assertIn("%%%% original patch: %s" % patchname, result.output)
+        self.assertIn("Notes (devtool):\n    original patch: %s" % patchname, result.output)
 
         # Configure the recipe to check that the git dependencies are correctly patched in cargo config
         bitbake('-c configure %s' % testrecipe)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 6d7fd17fbd..7972b4f822 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -937,14 +937,13 @@ def modify(args, config, basepath, workspace):
             seen_patches = []
             for branch in branches:
                 branch_patches[branch] = []
-                (stdout, _) = bb.process.run('git log devtool-base..%s' % branch, cwd=srctree)
-                for line in stdout.splitlines():
-                    line = line.strip()
-                    if line.startswith(oe.patch.GitApplyTree.patch_line_prefix):
-                        origpatch = line[len(oe.patch.GitApplyTree.patch_line_prefix):].split(':', 1)[-1].strip()
-                        if not origpatch in seen_patches:
-                            seen_patches.append(origpatch)
-                            branch_patches[branch].append(origpatch)
+                (stdout, _) = bb.process.run('git rev-list devtool-base..%s' % branch, cwd=srctree)
+                for sha1 in stdout.splitlines():
+                    notes = oe.patch.GitApplyTree.getNotes(srctree, sha1.strip())
+                    origpatch = notes.get(oe.patch.GitApplyTree.original_patch)
+                    if origpatch and origpatch not in seen_patches:
+                        seen_patches.append(origpatch)
+                        branch_patches[branch].append(origpatch)
 
         # Need to grab this here in case the source is within a subdirectory
         srctreebase = srctree
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index ef58523dc8..fa5b8ef3c7 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -261,7 +261,7 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
     revs = {}
     for path in paths:
         (stdout, _) = _run('git rev-parse HEAD', cwd=path)
-        revs[os.path.relpath(path,srctree)] = stdout.rstrip()
+        revs[os.path.relpath(path, srctree)] = stdout.rstrip()
 
     if no_patch:
         patches = oe.recipeutils.get_recipe_patches(crd)
@@ -272,17 +272,35 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
             _run('git checkout devtool-patched -b %s' % branch, cwd=path)
             (stdout, _) = _run('git branch --list devtool-override-*', cwd=path)
             branches_to_rebase = [branch] + stdout.split()
+            target_branch = revs[os.path.relpath(path, srctree)]
+
+            # There is a bug (or feature?) in git rebase where if a commit with
+            # a note is fully rebased away by being part of an old commit, the
+            # note is still attached to the old commit. Avoid this by making
+            # sure all old devtool related commits have a note attached to them
+            # (this assumes git config notes.rewriteMode is set to ignore).
+            (stdout, _) = __run('git rev-list devtool-base..%s' % target_branch)
+            for rev in stdout.splitlines():
+                if not oe.patch.GitApplyTree.getNotes(path, rev):
+                    oe.patch.GitApplyTree.addNote(path, rev, "dummy")
+
             for b in branches_to_rebase:
-                logger.info("Rebasing {} onto {}".format(b, revs[os.path.relpath(path,srctree)]))
+                logger.info("Rebasing {} onto {}".format(b, target_branch))
                 _run('git checkout %s' % b, cwd=path)
                 try:
-                    _run('git rebase %s' % revs[os.path.relpath(path, srctree)], cwd=path)
+                    _run('git rebase %s' % target_branch, cwd=path)
                 except bb.process.ExecutionError as e:
                     if 'conflict' in e.stdout:
                         logger.warning('Command \'%s\' failed:\n%s\n\nYou will need to resolve conflicts in order to complete the upgrade.' % (e.command, e.stdout.rstrip()))
                         _run('git rebase --abort', cwd=path)
                     else:
                         logger.warning('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
+
+            # Remove any dummy notes added above.
+            (stdout, _) = __run('git rev-list devtool-base..%s' % target_branch)
+            for rev in stdout.splitlines():
+                oe.patch.GitApplyTree.removeNote(path, rev, "dummy")
+
             _run('git checkout %s' % branch, cwd=path)
 
     if tmpsrctree:

Re: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches

[Ross Burton]

On 19 Feb 2024, at 01:28, Peter Kjellerstedt via lists.openembedded.org <peter.kjellerstedt=axis.com@lists.openembedded.org> wrote:
> +    @staticmethod
> +    def addNote(repo, ref, key, value=None):
> +        note = key + (": %s" % value if value else "")
> +        notes_ref = GitApplyTree.notes_ref
> +        runcmd(["git", "config", "notes.rewriteMode", "ignore"], repo)
> +        runcmd(["git", "config", "notes.displayRef", notes_ref, notes_ref], repo)
> +        runcmd(["git", "config", "notes.rewriteRef", notes_ref, notes_ref], repo)
> +        runcmd(["git", "notes", "--ref", notes_ref, "append", "-m", note, ref], repo)

It feels like the config calls could be done once when setting up the repository somehow?

> +                        # This loop, which is used to remove any line that
> +                        # starts with "%% original patch", is kept for backwards
> +                        # compatibility. If/when that compatibility is dropped,
> +                        # it can be replaced with code to just read the first
> +                        # line of the patch file to get the SHA-1, and the code
> +                        # below that writes the modified patch file can be
> +                        # replaced with a simple file move.

This is all that is needed for the new code to work correctly with an old worktree, right?

Ross

RE: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches

[Peter Kjellerstedt]

> -----Original Message-----
> From: Ross Burton <Ross.Burton@arm.com>
> Sent: den 19 februari 2024 13:56
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store
> the filenames for the patches
> 
> On 19 Feb 2024, at 01:28, Peter Kjellerstedt via lists.openembedded.org
> <peter.kjellerstedt=axis.com@lists.openembedded.org> wrote:
> > +    @staticmethod
> > +    def addNote(repo, ref, key, value=None):
> > +        note = key + (": %s" % value if value else "")
> > +        notes_ref = GitApplyTree.notes_ref
> > +        runcmd(["git", "config", "notes.rewriteMode", "ignore"], repo)
> > +        runcmd(["git", "config", "notes.displayRef", notes_ref, notes_ref], repo)
> > +        runcmd(["git", "config", "notes.rewriteRef", notes_ref, notes_ref], repo)
> > +        runcmd(["git", "notes", "--ref", notes_ref, "append", "-m", note, ref], repo)
> 
> It feels like the config calls could be done once when setting up the
> repository somehow?

While I do agree that would be better, the reason I did it like this is because 
there is nothing in this class that is related to setting up the repositories 
that are used while calling this function. Thus it felt wrong to rely on that 
the repository has been configured correctly beforehand for the function to 
work as intended.

> > +                        # This loop, which is used to remove any line that
> > +                        # starts with "%% original patch", is kept for backwards
> > +                        # compatibility. If/when that compatibility is dropped,
> > +                        # it can be replaced with code to just read the first
> > +                        # line of the patch file to get the SHA-1, and the code
> > +                        # below that writes the modified patch file can be
> > +                        # replaced with a simple file move.
> 
> This is all that is needed for the new code to work correctly with an old
> worktree, right?

Well, it relies on the fallback (i.e., the exception clause) in GitApplyTree.getNotes() 
that reads from the commit message in case a Git note cannot be found. Do you want me 
to extend the comment so that it includes that refence as well?

> 
> Ross

//Peter

Re: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches

[Ross Burton]

On 19 Feb 2024, at 15:03, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote:
>> This is all that is needed for the new code to work correctly with an old
>> worktree, right?
> 
> Well, it relies on the fallback (i.e., the exception clause) in GitApplyTree.getNotes() 
> that reads from the commit message in case a Git note cannot be found. Do you want me 
> to extend the comment so that it includes that refence as well?

No need, I was just checking that the new code would work on an old worktree using comments.

Ross

Re: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches

[Alexander Kanavin]

On Mon, 19 Feb 2024 at 02:28, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> +            target_branch = revs[os.path.relpath(path, srctree)]
> +
> +            # There is a bug (or feature?) in git rebase where if a commit with
> +            # a note is fully rebased away by being part of an old commit, the
> +            # note is still attached to the old commit. Avoid this by making
> +            # sure all old devtool related commits have a note attached to them
> +            # (this assumes git config notes.rewriteMode is set to ignore).
> +            (stdout, _) = __run('git rev-list devtool-base..%s' % target_branch)
> +            for rev in stdout.splitlines():
> +                if not oe.patch.GitApplyTree.getNotes(path, rev):
> +                    oe.patch.GitApplyTree.addNote(path, rev, "dummy")
> +

I've been wondering why devtool version upgrades in recipes that fetch
from git are taking significantly longer than they used to, and this
loop is basically the reason. It attaches a note to every commit
between the old and the new version tag in the upstream tree, and if
there's many such commits, it can take minutes, or hours. Things came
to a head in the most recent AUH run, which had tried to update
binutils, and was stopped on timeout after some 5 hours or so:

INFO: binutils-testsuite,binutils-cross-x86_64,binutils-crosssdk-x86_64-pokysdk-linux,binutils-cross-canadian-x86-64,binutils:
Running 'devtool upgrade' ...
command timed out: 16200 seconds without output running
[b'/srv/pokybuild/yocto-worker/auh/yocto-autobuilder-helper/scripts/run-config',
b'auh', b'/srv/pokybuild/yocto-worker/auh/build/build', b'master',
b'ssh://git@push.yoctoproject.org/poky', b'--sstateprefix', b'',
b'--buildappsrcrev', b'', b'--publish-dir', b'', b'--build-type',
b'quick', b'--workername', b'alma8-vk-2', b'--build-url',
b'https://autobuilder.yoctoproject.org/valkyrie/#/builders/38/builds/54',
b'--results-dir',
b'/srv/autobuilder/valkyrie.yocto.io/pub/non-release/20260215-30/testresults',
b'--quietlogging', b'--stepname', b'cmds', b'--phase', b'2'],
attempting to kill
process killed by signal 9

I'm not exactly sure what issue is being avoided here, but we can't do
it like that. I'm going to try and drop this code and see what breaks
in 'oe-selftest -r devtool', but if you can help out and come up with
the right fix, that would be very welcome.

Reproducer in oe-core master:
devtool upgrade systemd-systemctl-native -V 259.1 -S
6a36f1c7420dcb8e5176796676047f058466fe7a

Alex

RE: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 17 februari 2026 14:14
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org; Ross Burton <ross.burton@arm.com>; Richard Purdie <richard.purdie@linuxfoundation.org>
> Subject: Re: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches
> 
> On Mon, 19 Feb 2024 at 02:28, Peter Kjellerstedt <peter.kjellerstedt@axis.com> wrote:
> > +            target_branch = revs[os.path.relpath(path, srctree)]
> > +
> > +            # There is a bug (or feature?) in git rebase where if a commit with
> > +            # a note is fully rebased away by being part of an old commit, the
> > +            # note is still attached to the old commit. Avoid this by making
> > +            # sure all old devtool related commits have a note attached to them
> > +            # (this assumes git config notes.rewriteMode is set to ignore).
> > +            (stdout, _) = __run('git rev-list devtool-base..%s' % target_branch)
> > +            for rev in stdout.splitlines():
> > +                if not oe.patch.GitApplyTree.getNotes(path, rev):
> > +                    oe.patch.GitApplyTree.addNote(path, rev, "dummy")
> > +
> 
> I've been wondering why devtool version upgrades in recipes that fetch
> from git are taking significantly longer than they used to, and this
> loop is basically the reason. It attaches a note to every commit
> between the old and the new version tag in the upstream tree, and if
> there's many such commits, it can take minutes, or hours. Things came
> to a head in the most recent AUH run, which had tried to update
> binutils, and was stopped on timeout after some 5 hours or so:
> 
> INFO: binutils-testsuite,binutils-cross-x86_64,binutils-crosssdk-x86_64-pokysdk-linux,binutils-cross-canadian-x86-64,binutils:
> Running 'devtool upgrade' ...
> command timed out: 16200 seconds without output running
> [b'/srv/pokybuild/yocto-worker/auh/yocto-autobuilder-helper/scripts/run-config',
> b'auh', b'/srv/pokybuild/yocto-worker/auh/build/build', b'master',
> b'ssh://git@push.yoctoproject.org/poky', b'--sstateprefix', b'',
> b'--buildappsrcrev', b'', b'--publish-dir', b'', b'--build-type',
> b'quick', b'--workername', b'alma8-vk-2', b'--build-url',
> b'https://autobuilder.yoctoproject.org/valkyrie/#/builders/38/builds/54',
> b'--results-dir',
> b'/srv/autobuilder/valkyrie.yocto.io/pub/non-release/20260215-30/testresults',
> b'--quietlogging', b'--stepname', b'cmds', b'--phase', b'2'],
> attempting to kill
> process killed by signal 9
> 
> I'm not exactly sure what issue is being avoided here, but we can't do
> it like that. I'm going to try and drop this code and see what breaks
> in 'oe-selftest -r devtool', but if you can help out and come up with
> the right fix, that would be very welcome.
> 
> Reproducer in oe-core master:
> devtool upgrade systemd-systemctl-native -V 259.1 -S 6a36f1c7420dcb8e5176796676047f058466fe7a
> 
> Alex

This was two years ago, so unfortunately I do not remember the exact 
circumstances. I have tried to recreate the problem based on the comment 
in the code above, but I have not been able to do it. This may be due to 
the version of Git that I am using now, compared to what I had two years 
ago. However, I also tried it on my old computer that I used when I made 
the change above, and which I believe still has the same version of Git 
as it did back then. Unfortunately, I still was not able to reproduce 
the problem. :( So either I suck at writing comments describing problems, 
or I suck at understanding them later...

I also tried to figure out what (bad) consequences there would be if a 
note is attached to an old commit during the rebase, but I really can't 
come up with anything. At worst it would be a little annoying when doing 
git log. However, I can't see that it would affect the functionality of 
devtool in any way.

So based on the above, it is probably ok to remove the code that sets 
and removes the dummy notes.

//Peter

Re: [OE-core] [PATCHv2 5/5] lib/oe/patch: Use git notes to store the filenames for the patches

[Alexander Kanavin]

On Wed, 18 Feb 2026 at 03:18, Peter Kjellerstedt via
lists.openembedded.org
<peter.kjellerstedt=axis.com@lists.openembedded.org> wrote:
> This was two years ago, so unfortunately I do not remember the exact
> circumstances. I have tried to recreate the problem based on the comment
> in the code above, but I have not been able to do it. This may be due to
> the version of Git that I am using now, compared to what I had two years
> ago. However, I also tried it on my old computer that I used when I made
> the change above, and which I believe still has the same version of Git
> as it did back then. Unfortunately, I still was not able to reproduce
> the problem. :( So either I suck at writing comments describing problems,
> or I suck at understanding them later...
>
> I also tried to figure out what (bad) consequences there would be if a
> note is attached to an old commit during the rebase, but I really can't
> come up with anything. At worst it would be a little annoying when doing
> git log. However, I can't see that it would affect the functionality of
> devtool in any way.
>
> So based on the above, it is probably ok to remove the code that sets
> and removes the dummy notes.

Thanks for looking into it, Peter. I have meanwhile made a commit that
removes the code, and ran 'oe-selftest -r devtool' with it. There were
no failures.

So the next step is to do an AUH run:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/38/builds/58
and carefully review the resulting upgrade messages/patches for
correctness - either there will be a test case for what the dummy
notes were aiming to fix, or an additional confirmation that the code
is okay to remove, as the problem is either very niche, or
non-existent. Let's see what comes out of it.

Alex