[PATCH 1/2] devtool: upgrade: ignore changelogs from 3rd party

[PATCH 1/2] devtool: upgrade: ignore changelogs from 3rd party

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

Some upstream projects bundle vendored dependencies in their source
tree (e.g. nghttp2 ships third-party/mruby, which has its own
NEWS.md). The changelog extractor could mistake one of these
vendored changelogs for the recipe's own, misattributing unrelated
upstream changes to the package being upgraded.

Exclude paths under common vendoring directory names (third-party,
vendor, external, deps, etc.) from changelog candidates.

AI-Generated: Kiro with Claude Sonnet 5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 scripts/lib/devtool/upgrade.py | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 13d51bf952..495a5b8217 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -55,6 +55,13 @@ _CHANGELOG_BASENAMES = {
     'perldelta.pod',
 }
 
+# Path components indicating a bundled/vendored dependency rather than the
+# recipe's own source, so its changelog-like files should not be mistaken
+# for the recipe's own changes (e.g. third-party/mruby/NEWS.md in nghttp2).
+_VENDORED_PATH_RE = re.compile(
+    r'(^|/)(third[-_]party|vendor|vendored|external|extern|deps|3rdparty)(/|$)',
+    re.IGNORECASE)
+
 def _run(cmd, cwd=''):
     logger.debug("Running command %s> %s" % (cwd,cmd))
     return bb.process.run('%s' % cmd, cwd=cwd)
@@ -591,6 +598,9 @@ def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspac
     try:
         stdout, _ = _run('git diff --name-only %s %s' % (old_tag, new_tag), srctree)
         changed_files = [f.strip() for f in stdout.splitlines() if f.strip()]
+        # Exclude bundled/vendored dependencies; their changelogs are not
+        # relevant to this recipe's own version bump.
+        changed_files = [f for f in changed_files if not _VENDORED_PATH_RE.search(f)]
 
         # First pass: collect per-version release notes that changed
         # Matches files with a version number whose path suggests release notes

[PATCH 2/2] devtool: upgrade: diff git-log-style changelogs by commit hash

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

Files like nghttp2's ChangeLog are the literal output of `git log`,
regenerated wholesale on every release. New commits are prepended, so
every existing commit shifts position even though its content is
unchanged. A textual diff sees that shift as removed/added lines
instead of showing just the commits that were actually added.

Diff these files structurally instead: compare commit hashes between
the old and new file to tell moved commits from genuinely new ones,
and report the subject lines of the new commits. Falls back to the
existing line-based diff for changelogs that aren't git-log output.

AI-Generated: Kiro with Claude Sonnet 5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 scripts/lib/devtool/upgrade.py | 71 +++++++++++++++++++++++++++++-----
 1 file changed, 62 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 495a5b8217..0155578d68 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -589,6 +589,64 @@ def _resolve_rst_includes(content, srctree):
     return ''.join(result)
 
 
+_GIT_LOG_COMMIT_RE = re.compile(r'^commit ([0-9a-f]{7,40})\b.*$', re.MULTILINE)
+_GIT_LOG_HEADER_RE = re.compile(r'^(Merge|Author|AuthorDate|Commit|CommitDate):')
+
+def _diff_git_log_changelog(old_content, new_content):
+    """Structurally diff a ChangeLog file that is itself the output of
+    `git log` (e.g. nghttp2). Such files are regenerated wholesale on every
+    release: new commits are prepended, so every existing commit shifts
+    position even though its content is unchanged. A textual diff would
+    see that shift as removed/added lines instead of showing just the
+    commits that were actually added. Comparing commit hashes instead of
+    line content tells the two cases apart: hashes already present in
+    old_content are commits that merely moved, so they're skipped; any
+    other hash is a genuinely new commit. Returns the subject line of
+    each new commit as a single string, or None if either file doesn't
+    look like git log output."""
+    old_hashes = set(_GIT_LOG_COMMIT_RE.findall(old_content))
+    new_commits = _GIT_LOG_COMMIT_RE.split(new_content)[1:]  # [hash, block, hash, block, ...]
+    if not old_hashes or not new_commits:
+        return None
+
+    subjects = []
+    for commit_hash, block in zip(new_commits[0::2], new_commits[1::2]):
+        if commit_hash in old_hashes:
+            continue
+        message_lines = [line.strip() for line in block.splitlines()
+                          if line.strip() and not _GIT_LOG_HEADER_RE.match(line)]
+        # Skip the 'Merge pull request ...' line GitHub adds as the first
+        # message line of a merge commit; the actual change subject is the
+        # next line, and keeping the merge line would duplicate it.
+        subject = next((l for l in message_lines if not l.startswith('Merge pull request ')), None)
+        # Collapse adjacent duplicates: squash-merged PRs can produce
+        # consecutive commits with the same subject line.
+        if subject and (not subjects or subjects[-1] != subject):
+            subjects.append(subject)
+    return '\n'.join(subjects) if subjects else None
+
+
+def _changelog_file_candidate(srctree, old_tag, new_tag, fname):
+    """Return the changelog text to use for a single changelog-like file that
+    changed between old_tag and new_tag, or None if it has no usable
+    content. Tries a structural git-log diff first (for files like
+    nghttp2's ChangeLog that are `git log` output regenerated on every
+    release), falling back to the added lines of a plain-text diff."""
+    try:
+        old_file, _ = _run('git show %s' % shlex.quote('%s:%s' % (old_tag, fname)), srctree)
+        new_file, _ = _run('git show %s' % shlex.quote('%s:%s' % (new_tag, fname)), srctree)
+        candidate = _diff_git_log_changelog(old_file, new_file)
+        if candidate is not None:
+            return candidate
+    except bb.process.ExecutionError:
+        pass
+
+    diff_out, _ = _run('git diff %s %s -- %s' % (old_tag, new_tag, shlex.quote(fname)), srctree)
+    added_lines = [line[1:] for line in diff_out.splitlines()
+                   if line.startswith('+') and not line.startswith('+++')]
+    return '\n'.join(added_lines) if added_lines else None
+
+
 def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspace_path, is_git_source):
     """Extract changelog between old and new version using devtool git tags."""
     changelog_content = None
@@ -630,15 +688,10 @@ def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspac
             for fname in changed_files:
                 basename = os.path.basename(fname).lower()
                 if basename in _CHANGELOG_BASENAMES:
-                    diff_out, _ = _run('git diff %s %s -- %s' % (old_tag, new_tag, shlex.quote(fname)), srctree)
-                    if diff_out.strip():
-                        lines = [line[1:] for line in diff_out.splitlines()
-                                 if line.startswith('+') and not line.startswith('+++')]
-                        if lines:
-                            candidate = '\n'.join(lines)
-                            if not changelog_content or len(candidate) > len(changelog_content):
-                                changelog_content = candidate
-                                changelog_fname = fname
+                    candidate = _changelog_file_candidate(srctree, old_tag, new_tag, fname)
+                    if candidate and (not changelog_content or len(candidate) > len(changelog_content)):
+                        changelog_content = candidate
+                        changelog_fname = fname
     except bb.process.ExecutionError as e:
         logger.warning('Changelog file extraction failed: %s' % str(e))
 

Re: [OE-core] [PATCH 1/2] devtool: upgrade: ignore changelogs from 3rd party

[Richard Purdie]

On Mon, 2026-08-03 at 12:02 +0200, Daniel Turull via lists.openembedded.org wrote:
> From: Daniel Turull <daniel.turull@ericsson.com>
> 
> Some upstream projects bundle vendored dependencies in their source
> tree (e.g. nghttp2 ships third-party/mruby, which has its own
> NEWS.md). The changelog extractor could mistake one of these
> vendored changelogs for the recipe's own, misattributing unrelated
> upstream changes to the package being upgraded.
> 
> Exclude paths under common vendoring directory names (third-party,
> vendor, external, deps, etc.) from changelog candidates.
> 
> AI-Generated: Kiro with Claude Sonnet 5
> Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
> ---
>  scripts/lib/devtool/upgrade.py | 10 ++++++++++
>  1 file changed, 10 insertions(+)

Thanks for these. Unfortunately something in here caused the devtool tests to fail:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4274

Cheers,

Richard

Re: [OE-core] [PATCH 1/2] devtool: upgrade: ignore changelogs from 3rd party

[Daniel Turull]

On Mon, 2026-08-03 at 14:12 +0100, Richard Purdie wrote:
> On Mon, 2026-08-03 at 12:02 +0200, Daniel Turull via lists.openembedded.org wrote:
> > From: Daniel Turull <daniel.turull@ericsson.com>
> > 
> > Some upstream projects bundle vendored dependencies in their source
> > tree (e.g. nghttp2 ships third-party/mruby, which has its own
> > NEWS.md). The changelog extractor could mistake one of these
> > vendored changelogs for the recipe's own, misattributing unrelated
> > upstream changes to the package being upgraded.
> > 
> > Exclude paths under common vendoring directory names (third-party,
> > vendor, external, deps, etc.) from changelog candidates.
> > 
> > AI-Generated: Kiro with Claude Sonnet 5
> > Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
> > ---
> >  scripts/lib/devtool/upgrade.py | 10 ++++++++++
> >  1 file changed, 10 insertions(+)
> 
> Thanks for these. Unfortunately something in here caused the devtool tests to fail:
> 

Ok. I'll take a look and run the test locally. I just did happy testing with a couple of packages
and miss to run the devtool selftests.

Best regards,

Daniel
> 
> Cheers,
> 
> Richard