[PATCH v3 0/2] devtool: upgrade: improve changelog extraction for git-log-style ChangeLogs

[PATCH v3 0/2] devtool: upgrade: improve changelog extraction for git-log-style ChangeLogs

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

devtool upgrade's changelog extraction misattributed unrelated commits
when upgrading recipes like nghttp2, whose ChangeLog is the literal output
of `git log`, regenerated wholesale on every release.

Changes from v1:
- Header-skipping in the structural diff now relies on the blank line
  git log always inserts before the commit message, instead of
  enumerating header prefixes with a regex.
- The UTF-8 decode fix is split into its own patch and applied to both
  git-show call sites instead of just one.
- A single-use helper that only sequenced two operations with no reuse
  benefit has been inlined into its caller.
- The git-diff fallback is now guarded against UnicodeDecodeError too,
  so a bad byte in the diff hunk itself doesn't crash extraction.

Changes from v2:
- drop already merged patch.
- as suggested in the review use subprocess.run() and drop helper function

Daniel Turull (2):
  devtool: upgrade: call subprocess.run() instead of bb.process.run()
  devtool: upgrade: diff git-log-style changelogs by commit hash

 scripts/lib/devtool/upgrade.py | 68 +++++++++++++++++++++++++++++-----
 1 file changed, 58 insertions(+), 10 deletions(-)

[PATCH v3 1/2] devtool: upgrade: call subprocess.run() instead of bb.process.run()

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

bb.process.run() is intended for bitbake's own use: it wraps subprocess
with logging behaviour and always decodes command output as UTF-8. The
latter makes _run() raise UnicodeDecodeError on files containing
non-UTF-8 bytes (e.g. Latin-1 author names in a changelog), which
_extract_changelog() hits when reading release notes with `git show`.

Call subprocess.run() directly and decode with errors='replace' to
tolerate that. Keep raising bb.process.ExecutionError so the existing
callers and their error messages are unaffected.

AI-Generated: Kiro with Claude Sonnet 5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 scripts/lib/devtool/upgrade.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 495a5b8217..d74ffcf534 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -11,6 +11,7 @@ import sys
 import re
 import shlex
 import shutil
+import subprocess
 import tempfile
 import logging
 import argparse
@@ -64,7 +65,11 @@ _VENDORED_PATH_RE = re.compile(
 
 def _run(cmd, cwd=''):
     logger.debug("Running command %s> %s" % (cwd,cmd))
-    return bb.process.run('%s' % cmd, cwd=cwd)
+    result = subprocess.run(cmd, cwd=cwd or None, shell=True, capture_output=True,
+                            text=True, errors='replace')
+    if result.returncode != 0:
+        raise bb.process.ExecutionError(cmd, result.returncode, result.stdout, result.stderr)
+    return (result.stdout, result.stderr)
 
 def _get_srctree(tmpdir):
     srctree = tmpdir

[PATCH v3 2/2] devtool: upgrade: diff git-log-style changelogs by commit hash

[daniel.turull]

From: Daniel Turull <daniel.turull@ericsson.com>

Files like nghttp2's ChangeLog are the literal output of `git log`,
regenerated wholesale on every release. New commits are prepended, so
every existing commit shifts position even though its content is
unchanged. A textual diff sees that shift as removed/added lines
instead of showing just the commits that were actually added.

Diff these files structurally instead: compare commit hashes between
the old and new file to tell moved commits from genuinely new ones,
and report the subject lines of the new commits. Falls back to the
existing line-based diff for changelogs that aren't git-log output.

Tested with: oe-selftest -r devtool.DevtoolUpgradeTests

AI-Generated: Kiro with Claude Sonnet 5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
---
 scripts/lib/devtool/upgrade.py | 61 +++++++++++++++++++++++++++++-----
 1 file changed, 52 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index d74ffcf534..523acf4250 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -594,6 +594,43 @@ def _resolve_rst_includes(content, srctree):
     return ''.join(result)
 
 
+_GIT_LOG_COMMIT_RE = re.compile(r'^commit ([0-9a-f]{7,40})\b.*$', re.MULTILINE)
+
+def _diff_git_log_changelog(old_content, new_content):
+    """Diff a ChangeLog that is itself `git log` output (e.g. nghttp2),
+    regenerated wholesale on every release. New commits get prepended, so
+    old ones shift position even though unchanged, which a textual diff
+    would wrongly show as removed/added lines. Compare commit hashes
+    instead: hashes already in old_content are just moved, anything else
+    is new. Returns the new commits' subject lines as one string, or
+    None if either file doesn't look like git log output."""
+    old_hashes = set(_GIT_LOG_COMMIT_RE.findall(old_content))
+    new_commits = _GIT_LOG_COMMIT_RE.split(new_content)[1:]  # [hash, block, hash, block, ...]
+    if not old_hashes or not new_commits:
+        return None
+
+    subjects = []
+    for commit_hash, block in zip(new_commits[0::2], new_commits[1::2]):
+        if commit_hash in old_hashes:
+            continue
+        # `git log` separates headers (Author:, Date:, etc.) from the
+        # commit message with a blank line; drop the leading blank line
+        # left over from splitting on "commit <hash>", then skip the
+        # headers up to the next blank line to get just the message.
+        lines = block.splitlines()
+        while lines and not lines[0].strip():
+            lines.pop(0)
+        blank_at = next((i for i, l in enumerate(lines) if not l.strip()), len(lines))
+        message_lines = [l.strip() for l in lines[blank_at:] if l.strip()]
+        # Skip the 'Merge pull request ...' line GitHub adds as the first
+        # message line of a merge commit; the actual change subject is the
+        # next line, and keeping the merge line would duplicate it.
+        subject = next((l for l in message_lines if not l.startswith('Merge pull request ')), None)
+        if subject and (not subjects or subjects[-1] != subject):
+            subjects.append(subject)
+    return '\n'.join(subjects) if subjects else None
+
+
 def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspace_path, is_git_source):
     """Extract changelog between old and new version using devtool git tags."""
     changelog_content = None
@@ -635,15 +672,21 @@ def _extract_changelog(srctree, pn, old_ver, new_ver, old_tag, new_tag, workspac
             for fname in changed_files:
                 basename = os.path.basename(fname).lower()
                 if basename in _CHANGELOG_BASENAMES:
-                    diff_out, _ = _run('git diff %s %s -- %s' % (old_tag, new_tag, shlex.quote(fname)), srctree)
-                    if diff_out.strip():
-                        lines = [line[1:] for line in diff_out.splitlines()
-                                 if line.startswith('+') and not line.startswith('+++')]
-                        if lines:
-                            candidate = '\n'.join(lines)
-                            if not changelog_content or len(candidate) > len(changelog_content):
-                                changelog_content = candidate
-                                changelog_fname = fname
+                    candidate = None
+                    try:
+                        old_file, _ = _run('git show %s' % shlex.quote('%s:%s' % (old_tag, fname)), srctree)
+                        new_file, _ = _run('git show %s' % shlex.quote('%s:%s' % (new_tag, fname)), srctree)
+                        candidate = _diff_git_log_changelog(old_file, new_file)
+                    except bb.process.ExecutionError:
+                        pass
+                    if candidate is None:
+                        diff_out, _ = _run('git diff %s %s -- %s' % (old_tag, new_tag, shlex.quote(fname)), srctree)
+                        added_lines = [line[1:] for line in diff_out.splitlines()
+                                       if line.startswith('+') and not line.startswith('+++')]
+                        candidate = '\n'.join(added_lines) if added_lines else None
+                    if candidate and (not changelog_content or len(candidate) > len(changelog_content)):
+                        changelog_content = candidate
+                        changelog_fname = fname
     except bb.process.ExecutionError as e:
         logger.warning('Changelog file extraction failed: %s' % str(e))