Openembedded Bitbake Development
 help / color / mirror / Atom feed
From: Savvas Etairidis <setairidis@gmail.com>
To: bitbake-devel@lists.openembedded.org
Cc: Paul Barker <paul@pbarker.dev>,
	Jhonata Poma-Hansen <jhonata.poma@gmail.com>,
	Savvas Etairidis <setairidis@gmail.com>
Subject: [PATCH v9] fetch/local: verify checksums for file:// urls
Date: Sat,  7 Nov 2026 21:23:48 +0100	[thread overview]
Message-ID: <20261107202348.29338-1-setairidis@gmail.com> (raw)
In-Reply-To: <344401.1794082642467075956@lists.openembedded.org>

[YOCTO #9993]

A checksum given in a file:// url (e.g. UNINATIVE_CHECKSUM when
UNINATIVE_URL points at a local file://) was never verified, because
Local disables the donestamp and so verify_checksum() was never called.
Enable the donestamp in Local.urldata_init() for file:// urls that carry
a checksum parameter.

Based on a patch by Jhonata Poma-Hansen.

AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Savvas Etairidis <setairidis@gmail.com>

---
changes in v9:
- Moved the donestamp decision from FetchData.__init__() into
  Local.urldata_init(), with a Local.checksum_expected() helper that
  checks the url parameters (the *_expected attributes are not yet set
  when urldata_init() runs).
- Made rename_bad_checksum() a FetchMethod method, overridden in Local,
  instead of special-casing Local in the base fetcher code.
- Tested with BB_SKIP_NETTESTS=yes bin/bitbake-selftest bb.tests.fetch

changes in v8:
- Dropped the local-checksums stamps and re-used the regular donestamp,
  enabled only for file:// urls with a checksum. local.py is unchanged
  again.
- Tests reduced to a checksum match and a mismatch test.
- Tested with BB_SKIP_NETTESTS=yes bin/bitbake-selftest bb.tests.fetch

changes in v7:
- We shouldn't assume that anything coming from a file:// URL is
  "typically small", or that it is on fast local storage (it could be on a
  HDD array accessed over NFS).
* Rework the code to avoid re-hashing the file on every fetch, and
  instead only re-hash it when the checksum is actually expected to be verified.
- You don't need to mention this if it didn't result in actual change.
* Wont do in the future.
- Such as? This is a completely different approach so it's useful to know
  which bits of feedback were actually relevant and which were made
  obsolete by the change in approach.
* Hoped I did better this time.
- What does "edited with an old mtime" mean?
* In that test the file attributes are modified to have an old mtime,
  which is a corner case that was not handled correctly before.
  This test was removed in v6 because the new approach doesn't have
  that issue anymore.
- Why complicate this by using different hash algorithms for the different
  tests?
  The test wants to cover the case where we have a valid md5sum but an
  invalid sha256sum or a invalid md5sum but a valid sha256sum.
- Added a test to covern new functionality.

changes in v6:
- Rebased on top of master
- Fixed issues reported by reviewer
- Check the file directly in Local.verify_donestamp(), via a new
  FetchData.verify_checksum_if_expected() helper, instead of enabling
  the donestamp. Fixes stale checksums for files edited with an old
  mtime and DL_DIR stamp/lock path collisions
- Drop the rename_bad_checksum() refactor, no longer needed
- Document the cost of re-hashing on every fetch
- Tests: use a correct md5sum and a wrong sha256sum in the mismatch
  test, add a test for a file edited with an old mtime, and check that
  nothing is written to DL_DIR
- Tested with BB_SKIP_NETTESTS=yes bin/bitbake-selftest bb.tests.fetch

changes in v5:
- Rebased on top of master, added changelog entry

changes in v4:
- Rebased on top of master

changes in v3:
- Cherry-picked the patch from Jhonata Poma-Hansen

---
---
 lib/bb/fetch/__init__.py | 41 ++++++++++++++++++++--------------------
 lib/bb/fetch/local.py    | 20 +++++++++++++++++++-
 lib/bb/fetch/wget.py     |  2 +-
 lib/bb/tests/fetch.py    | 30 +++++++++++++++++++++++++++++
 4 files changed, 70 insertions(+), 23 deletions(-)

diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index eb3483a48..d339ac1da 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -742,7 +742,7 @@ def verify_donestamp(ud, d, origud=None):
         logger.warning("Checksum mismatch for local file %s\n"
                        "Cleaning and trying again." % ud.localpath)
         if os.path.exists(ud.localpath):
-            rename_bad_checksum(ud, e.checksum)
+            ud.method.rename_bad_checksum(ud, e.checksum)
         bb.utils.remove(ud.donestamp)
     return False
 
@@ -775,7 +775,7 @@ def update_stamp(ud, d):
             logger.warning("Checksum mismatch for local file %s\n"
                            "Cleaning and trying again." % ud.localpath)
             if os.path.exists(ud.localpath):
-                rename_bad_checksum(ud, e.checksum)
+                ud.method.rename_bad_checksum(ud, e.checksum)
             bb.utils.remove(ud.donestamp)
             raise
 
@@ -1079,23 +1079,6 @@ def build_mirroruris(origud, mirrors, ld):
 
     return uris, uds
 
-def rename_bad_checksum(ud, suffix, localpath=None):
-    """
-    Renames files to have suffix from parameter
-    """
-
-    if ud.localpath is None:
-        return
-
-    if localpath is None:
-        localpath = ud.localpath
-
-    new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
-    bb.warn("Renaming %s to %s" % (localpath, new_localpath))
-    if not bb.utils.movefile(localpath, new_localpath):
-        bb.warn("Renaming %s to %s failed, grep movefile in log.do_fetch to see why" % (localpath, new_localpath))
-
-
 def try_mirror_url(fetch, origud, ud, ld, check = False):
     # Return of None or a value means we're finished
     # False means try another url
@@ -1166,7 +1149,7 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
             logger.warning("Mirror checksum failure for url %s (original url: %s)\nCleaning and trying again." % (ud.url, origud.url))
             logger.warning(str(e))
             if os.path.exists(ud.localpath):
-                rename_bad_checksum(ud, e.checksum)
+                ud.method.rename_bad_checksum(ud, e.checksum)
         elif isinstance(e, NoChecksumError):
             raise
         else:
@@ -1482,6 +1465,22 @@ class FetchMethod(object):
         """
         return True
 
+    def rename_bad_checksum(self, ud, suffix, localpath=None):
+        """
+        Renames files to have suffix from parameter
+        """
+
+        if ud.localpath is None:
+            return
+
+        if localpath is None:
+            localpath = ud.localpath
+
+        new_localpath = "%s_bad-checksum_%s" % (ud.localpath, suffix)
+        bb.warn("Renaming %s to %s" % (localpath, new_localpath))
+        if not bb.utils.movefile(localpath, new_localpath):
+            bb.warn("Renaming %s to %s failed, grep movefile in log.do_fetch to see why" % (localpath, new_localpath))
+
     def verify_donestamp(self, ud, d):
         """
         Verify the donestamp file
@@ -1946,7 +1945,7 @@ class Fetch(object):
                             logger.warning("Checksum failure encountered with download of %s - will attempt other sources if available" % u)
                             logger.debug(str(e))
                             if os.path.exists(ud.localpath):
-                                rename_bad_checksum(ud, e.checksum)
+                                ud.method.rename_bad_checksum(ud, e.checksum)
                         elif isinstance(e, NoChecksumError):
                             raise
                         else:
diff --git a/lib/bb/fetch/local.py b/lib/bb/fetch/local.py
index dfc5b564c..0c36ec39d 100644
--- a/lib/bb/fetch/local.py
+++ b/lib/bb/fetch/local.py
@@ -31,11 +31,29 @@ class Local(FetchMethod):
         # We don't set localfile as for this fetcher the file is already local!
         ud.basename = os.path.basename(ud.path)
         ud.basepath = ud.path
-        ud.needdonestamp = False
+        # Only use a donestamp when a checksum is given in the url, so that
+        # verify_checksum() gets called for the file
+        ud.needdonestamp = self.checksum_expected(ud)
         if "*" in ud.path:
             raise bb.fetch.ParameterError("file:// urls using globbing are no longer supported. Please place the files in a directory and reference that instead.", ud.url)
         return
 
+    def checksum_expected(self, ud):
+        """
+        Is any checksum given in the url parameters?
+        """
+        for checksum_id in bb.fetch.CHECKSUM_LIST:
+            if "%ssum" % checksum_id in ud.parm or \
+               "%s.%ssum" % (ud.name, checksum_id) in ud.parm:
+                return True
+        return False
+
+    def rename_bad_checksum(self, ud, suffix, localpath=None):
+        """
+        A file:// url refers to the user's own file, leave it in place
+        """
+        return
+
     def localpath(self, urldata, d):
         """
         Return the local filename of a given url assuming a successful fetch.
diff --git a/lib/bb/fetch/wget.py b/lib/bb/fetch/wget.py
index 0411a6740..a4eb9bf98 100644
--- a/lib/bb/fetch/wget.py
+++ b/lib/bb/fetch/wget.py
@@ -181,7 +181,7 @@ class Wget(FetchMethod):
         except bb.fetch.ChecksumError as e:
             # The download is resumed with --continue, so a bad file left in place
             # would be reused by the mirrors and by every later attempt
-            bb.fetch.rename_bad_checksum(ud, e.checksum, localpath=localpath)
+            self.rename_bad_checksum(ud, e.checksum, localpath=localpath)
             raise
 
         # Remove the ".tmp" and move the file into position atomically
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 0f29cdf11..3c518805c 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -876,6 +876,36 @@ class FetcherLocalTest(FetcherTest):
             with self.subTest(striplevel=repr(value)):
                 self.assertInvalidStriplevel(value)
 
+    def test_local_checksum_match(self):
+        # Correct sha256 must run verify_checksum to completion; donestamp
+        # lands in DL_DIR.
+        import hashlib
+        content = b"file:// checksum match test\n"
+        with open(os.path.join(self.localsrcdir, 'sumfile'), 'wb') as f:
+            f.write(content)
+        good = hashlib.sha256(content).hexdigest()
+        fetcher = bb.fetch.Fetch(['file://sumfile;sha256sum=' + good], self.d)
+        ud = fetcher.ud[fetcher.urls[0]]
+        self.assertTrue(ud.needdonestamp)
+        fetcher.download()
+        self.assertTrue(os.path.exists(ud.donestamp))
+
+    def test_local_checksum_mismatch(self):
+        # Bad sha256 raises ChecksumError without renaming the user's
+        # source file to the <localpath>_bad-checksum_<sha> sibling.
+        content = b"file:// checksum mismatch test\n"
+        srcpath = os.path.join(self.localsrcdir, 'sumfile')
+        with open(srcpath, 'wb') as f:
+            f.write(content)
+        bad = "0" * 64
+        fetcher = bb.fetch.Fetch(['file://sumfile;sha256sum=' + bad], self.d)
+        with self.assertRaises(bb.fetch2.FetchError):
+            fetcher.download()
+        self.assertTrue(os.path.exists(srcpath))
+        with open(srcpath, 'rb') as f:
+            self.assertEqual(f.read(), content)
+        self.assertFalse(os.path.exists(srcpath + '_bad-checksum_' + bad))
+
     def dummyGitTest(self, suffix):
         # Create dummy local Git repo
         src_dir = tempfile.mkdtemp(dir=self.tempdir,
-- 
2.53.0


      reply	other threads:[~2026-11-07 20:23 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-04-30 14:25 [PATCH] fetch2/local: verify checksums for file:// urls Jhonata Poma-Hansen
2026-05-08 14:13 ` [bitbake-devel] " Paul Barker
2026-05-11 14:58   ` [PATCH v2] " Jhonata Poma-Hansen
2026-05-11 15:01     ` Jhonata Poma-Hansen
2026-09-23 19:44     ` [PATCH v3] " Savvas Etairidis
2026-09-24  6:36       ` [bitbake-devel] " Antonin Godard
2026-09-24 20:45         ` [PATCH v4] fetch/local: " Savvas Etairidis
2026-09-28  9:30           ` Paul Barker
2026-09-28 20:59             ` [PATCH v5] " Savvas Etairidis
2026-09-28 21:02               ` Sawas Etairidis
2026-09-29  8:22                 ` Jhonata Poma-Hansen
2026-09-30 10:51               ` Paul Barker
2026-09-30 21:21                 ` [PATCH v6] " Savvas Etairidis
2026-10-01  8:32                   ` Paul Barker
2026-10-01 20:53                     ` [PATCH v7] " Savvas Etairidis
2026-10-02  8:50                       ` Paul Barker
2026-10-02 20:48                         ` Sawas Etairidis
2026-10-02 20:50                           ` [PATCH v8] " Savvas Etairidis
2026-10-05 14:36                             ` Paul Barker
2026-10-06 16:40                               ` Sawas Etairidis
2026-11-07 20:17                                 ` Savvas Etairidis
2026-11-07 20:23                                   ` Savvas Etairidis [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261107202348.29338-1-setairidis@gmail.com \
    --to=setairidis@gmail.com \
    --cc=bitbake-devel@lists.openembedded.org \
    --cc=jhonata.poma@gmail.com \
    --cc=paul@pbarker.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox