From: Savvas Etairidis <setairidis@gmail.com>
To: bitbake-devel@lists.openembedded.org
Cc: Jhonata Poma-Hansen <jhonata.poma@gmail.com>,
Paul Barker <paul@pbarker.dev>,
Savvas Etairidis <setairidis@gmail.com>
Subject: [PATCH v7] fetch/local: verify checksums for file:// urls
Date: Thu, 1 Oct 2026 22:53:44 +0200 [thread overview]
Message-ID: <20261001205344.99638-1-setairidis@gmail.com> (raw)
In-Reply-To: <abf12d51916cac8780914605168243c54494ca32.camel@pbarker.dev>
[YOCTO #9993]
A checksum given in a file:// url (e.g. UNINATIVE_CHECKSUM when
UNINATIVE_URL points at a local file://) was never verified, because
Local disables the donestamp and so verify_checksum() was never called.
Override verify_donestamp() in Local to check the file against those
checksums. A mismatch raises ChecksumError before the rename/mirror
handling, so the user's file is never renamed.
Based on a patch by Jhonata Poma-Hansen.
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Savvas Etairidis <setairidis@gmail.com>
---
changes in v7:
- We shouldn't assume that anything coming from a file:// URL is
"typically small", or that it is on fast local storage (it could be on a
HDD array accessed over NFS).
* Rework the code to avoid re-hashing the file on every fetch, and
instead only re-hash it when the checksum is actually expected to be verified.
- You don't need to mention this if it didn't result in actual change.
* Wont do in the future.
- Such as? This is a completely different approach so it's useful to know
which bits of feedback were actually relevant and which were made
obsolete by the change in approach.
* Hoped I did better this time.
- What does "edited with an old mtime" mean?
* In that test the file attributes are modified to have an old mtime,
which is a corner case that was not handled correctly before.
This test was removed in v6 because the new approach doesn't have
that issue anymore.
- Why complicate this by using different hash algorithms for the different
tests?
The test wants to cover the case where we have a valid md5sum but an
invalid sha256sum or a invalid md5sum but a valid sha256sum.
- Added a test to covern new functionality.
changes in v6:
- Rebased on top of master
- Fixed issues reported by reviewer
- Check the file directly in Local.verify_donestamp(), via a new
FetchData.verify_checksum_if_expected() helper, instead of enabling
the donestamp. Fixes stale checksums for files edited with an old
mtime and DL_DIR stamp/lock path collisions
- Drop the rename_bad_checksum() refactor, no longer needed
- Document the cost of re-hashing on every fetch
- Tests: use a correct md5sum and a wrong sha256sum in the mismatch
test, add a test for a file edited with an old mtime, and check that
nothing is written to DL_DIR
- Tested with BB_SKIP_NETTESTS=yes bin/bitbake-selftest bb.tests.fetch
changes in v5:
- Rebased on top of master, added changelog entry
changes in v4:
- Rebased on top of master
changes in v3:
- Cherry-picked the patch from Jhonata Poma-Hansen
---
---
lib/bb/fetch/__init__.py | 9 ++++
lib/bb/fetch/local.py | 82 +++++++++++++++++++++++++++++
lib/bb/tests/fetch.py | 109 +++++++++++++++++++++++++++++++++++++++
3 files changed, 200 insertions(+)
diff --git a/lib/bb/fetch/__init__.py b/lib/bb/fetch/__init__.py
index fb6278439..d74c20df4 100644
--- a/lib/bb/fetch/__init__.py
+++ b/lib/bb/fetch/__init__.py
@@ -1413,6 +1413,15 @@ class FetchData(object):
self.donestamp = basepath + '.done'
self.lockfile = basepath + '.lock'
+ def checksum_expected(self):
+ """
+ Is any checksum given for this url?
+ """
+ for checksum_id in CHECKSUM_LIST:
+ if getattr(self, "%s_expected" % checksum_id):
+ return True
+ return False
+
def setup_revisions(self, d):
self.revision = srcrev_internal_helper(self, d, self.name)
diff --git a/lib/bb/fetch/local.py b/lib/bb/fetch/local.py
index dfc5b564c..ca9f91497 100644
--- a/lib/bb/fetch/local.py
+++ b/lib/bb/fetch/local.py
@@ -14,6 +14,8 @@ BitBake build tools.
#
import os
+import hashlib
+import json
import urllib.request, urllib.parse, urllib.error
import bb
import bb.utils
@@ -36,6 +38,82 @@ class Local(FetchMethod):
raise bb.fetch.ParameterError("file:// urls using globbing are no longer supported. Please place the files in a directory and reference that instead.", ud.url)
return
+ def checksum_stamp(self, ud, d):
+ """
+ Return the stamp file caching the checksums of ud.localpath, or None.
+ """
+ dldir = d.getVar("DL_DIR")
+ if not dldir or not ud.localpath:
+ return None
+ path = os.path.abspath(ud.localpath)
+ return os.path.join(dldir, "local-checksums",
+ hashlib.sha256(path.encode("utf-8")).hexdigest())
+
+ def checksum_filestat(self, st):
+ """
+ Return a list of the file's stat information that is used to detect changes.
+ """
+ return [st.st_ino, st.st_size, st.st_mtime_ns, st.st_ctime_ns]
+
+ def read_checksum_stamp(self, ud, d, st):
+ """
+ Return the checksums cached in the stamp file for ud.localpath, or {}
+ if there are none or the file has changed since they were computed.
+ """
+ stamp = self.checksum_stamp(ud, d)
+ if not stamp:
+ return {}
+ try:
+ with open(stamp) as f:
+ cached = json.load(f)
+ if cached["stat"] == self.checksum_filestat(st) and isinstance(cached["checksums"], dict):
+ return cached["checksums"]
+ except (OSError, ValueError, KeyError, TypeError):
+ pass
+ return {}
+
+ def write_checksum_stamp(self, ud, d, st, checksums):
+ """
+ Cache the checksums computed for ud.localpath, which had the stat
+ result st before it was hashed, in the stamp file.
+ """
+ stamp = self.checksum_stamp(ud, d)
+ if not stamp:
+ return
+
+ # Written via rename as no lock is held for file:// urls. Errors
+ # aren't fatal, the file is just hashed again next time.
+ tmpstamp = "%s.%d" % (stamp, os.getpid())
+ try:
+ bb.utils.mkdirhier(os.path.dirname(stamp))
+ with open(tmpstamp, "w") as f:
+ json.dump({"path": ud.localpath, "stat": self.checksum_filestat(st), "checksums": checksums}, f)
+ os.rename(tmpstamp, stamp)
+ except OSError:
+ try:
+ os.unlink(tmpstamp)
+ except OSError:
+ pass
+
+ def verify_donestamp(self, ud, d):
+ # file:// urls have no donestamp; verify any checksum given in the url
+ # against the file itself. A missing file is reported by download()
+ # with the searched paths, so there's nothing to check here. A mismatch
+ # raises ChecksumError rather than returning False, as there's nothing
+ # to re-download.
+ if not ud.checksum_expected():
+ return True
+ try:
+ st = os.stat(ud.localpath)
+ except OSError:
+ return True
+
+ precomputed = self.read_checksum_stamp(ud, d, st)
+ checksums = bb.fetch.verify_checksum(ud, d, precomputed)
+ if checksums and checksums != precomputed:
+ self.write_checksum_stamp(ud, d, st, checksums)
+ return True
+
def localpath(self, urldata, d):
"""
Return the local filename of a given url assuming a successful fetch.
@@ -87,5 +165,9 @@ class Local(FetchMethod):
return False
def clean(self, urldata, d):
+ # The file is the user's; only remove our cached checksums for it
+ stamp = self.checksum_stamp(urldata, d)
+ if stamp:
+ bb.utils.remove(stamp)
return
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 71b0a63fe..cbb47849c 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -21,6 +21,7 @@ import subprocess
import json
import tarfile
import threading
+import time
from bb.fetch import URI
import bb
import bb.utils
@@ -876,6 +877,114 @@ class FetcherLocalTest(FetcherTest):
with self.subTest(striplevel=repr(value)):
self.assertInvalidStriplevel(value)
+ def writeSumfile(self, content, dirname=None):
+ path = os.path.join(dirname or self.localsrcdir, 'sumfile')
+ with open(path, 'wb') as f:
+ f.write(content)
+ return path, hashlib.sha256(content).hexdigest()
+
+ def localChecksumStamps(self):
+ stampdir = os.path.join(self.dldir, 'local-checksums')
+ return os.listdir(stampdir) if os.path.exists(stampdir) else []
+
+ def test_local_checksum_match(self):
+ srcpath, good = self.writeSumfile(b"file:// checksum match test\n")
+ fetcher = bb.fetch.Fetch(['file://sumfile;sha256sum=' + good], self.d)
+ fetcher.download()
+ # Only the checksum stamp should be created in DL_DIR
+ self.assertEqual(os.listdir(self.dldir), ['local-checksums'])
+ stamps = self.localChecksumStamps()
+ self.assertEqual(len(stamps), 1)
+ # The stamp must follow the umask, so a shared DL_DIR can re-use it
+ umask = os.umask(0)
+ os.umask(umask)
+ mode = os.stat(os.path.join(self.dldir, 'local-checksums', stamps[0])).st_mode
+ self.assertEqual(mode & 0o777, 0o666 & ~umask)
+ # Cleaning removes the stamp but not the user's file
+ fetcher.clean()
+ self.assertEqual(self.localChecksumStamps(), [])
+ self.assertTrue(os.path.exists(srcpath))
+
+ def test_local_checksum_no_checksum_no_stamp(self):
+ self.writeSumfile(b"file:// no checksum test\n")
+ bb.fetch.Fetch(['file://sumfile'], self.d).download()
+ self.assertEqual(os.listdir(self.dldir), [])
+
+ def test_local_checksum_stamp_write_failure(self):
+ # A failure to write the stamp isn't fatal and leaves nothing behind
+ srcpath, good = self.writeSumfile(b"file:// checksum stamp failure test\n")
+ with unittest.mock.patch('os.rename', side_effect=OSError("rename failed")):
+ bb.fetch.Fetch(['file://sumfile;sha256sum=' + good], self.d).download()
+ self.assertEqual(self.localChecksumStamps(), [])
+
+ def test_local_checksum_unreachable_file(self):
+ # A path below a regular file can't be stat()ed; this should be
+ # reported as a missing file, not as an OSError
+ srcpath, good = self.writeSumfile(b"file:// checksum unreachable test\n")
+ fetcher = bb.fetch.Fetch(['file://%s/sub;sha256sum=%s' % (srcpath, good)], self.d)
+ with self.assertRaises(bb.fetch.FetchError):
+ fetcher.download()
+
+ def test_local_checksum_unchanged_not_rehashed(self):
+ stampdir, good = self.writeSumfile(b"file:// checksum rehash test\n")
+ url = 'file://sumfile;sha256sum=' + good
+ with unittest.mock.patch('bb.utils.sha256_file',
+ wraps=bb.utils.sha256_file) as sha256_file:
+ bb.fetch.Fetch([url], self.d).download()
+ self.assertEqual(sha256_file.call_count, 1)
+ # A later fetch re-uses the checksums stored in the stamp
+ bb.fetch.Fetch([url], self.d).download()
+ self.assertEqual(sha256_file.call_count, 1)
+ # A wrong checksum in the url is still caught without re-hashing
+ with self.assertRaises(bb.fetch.ChecksumError):
+ bb.fetch.Fetch(['file://sumfile;sha256sum=' + "0" * 64], self.d).download()
+ self.assertEqual(sha256_file.call_count, 1)
+
+ def test_local_checksum_modified_keeping_mtime(self):
+ srcpath, good = self.writeSumfile(b"file:// checksum original content\n")
+ url = 'file://sumfile;sha256sum=' + good
+ bb.fetch.Fetch([url], self.d).download()
+
+ # Replace the content in place with the same size and restore the
+ # exact original mtime, as rsync -a or touch -r could; only ctime
+ # changes. Filesystem timestamps may only advance once per timer
+ # tick, so wait for longer than that first.
+ st = os.stat(srcpath)
+ time.sleep(0.05)
+ with open(srcpath, 'wb') as f:
+ f.write(b"file:// checksum modified content\n")
+ os.utime(srcpath, ns=(st.st_atime_ns, st.st_mtime_ns))
+ with self.assertRaises(bb.fetch.ChecksumError):
+ bb.fetch.Fetch([url], self.d).download()
+ self.assertTrue(os.path.exists(srcpath))
+
+ def test_local_checksum_mismatch(self):
+ content = b"file:// checksum mismatch test\n"
+ srcpath, _ = self.writeSumfile(content)
+ md5 = hashlib.md5(content).hexdigest()
+ # A correct checksum must not hide a wrong one
+ url = 'file://sumfile;md5sum=%s;sha256sum=%s' % (md5, "0" * 64)
+ fetcher = bb.fetch.Fetch([url], self.d)
+ with self.assertRaises(bb.fetch.ChecksumError) as cm:
+ fetcher.download()
+ self.assertIn("has sha256 checksum", str(cm.exception))
+ # The user's file must not be renamed to *_bad-checksum_*
+ self.assertTrue(os.path.exists(srcpath))
+ self.assertEqual(self.localChecksumStamps(), [])
+
+ def test_local_checksum_same_name_different_dirs(self):
+ # Files with the same name must not share a stamp
+ sums = {}
+ for name in ('a', 'b'):
+ dirname = os.path.join(self.localsrcdir, 'sumdir-' + name)
+ os.mkdir(dirname)
+ sums[name] = self.writeSumfile(b"file:// checksum dir %s\n" % name.encode(), dirname)
+ for srcpath, good in sums.values():
+ bb.fetch.Fetch(['file://%s;sha256sum=%s' % (srcpath, good)], self.d).download()
+ self.assertEqual(len(self.localChecksumStamps()), 2)
+ with self.assertRaises(bb.fetch.ChecksumError):
+ bb.fetch.Fetch(['file://%s;sha256sum=%s' % (sums['a'][0], sums['b'][1])], self.d).download()
+
def dummyGitTest(self, suffix):
# Create dummy local Git repo
src_dir = tempfile.mkdtemp(dir=self.tempdir,
--
2.34.1
next prev parent reply other threads:[~2026-10-01 20:53 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-30 14:25 [PATCH] fetch2/local: verify checksums for file:// urls Jhonata Poma-Hansen
2026-05-08 14:13 ` [bitbake-devel] " Paul Barker
2026-05-11 14:58 ` [PATCH v2] " Jhonata Poma-Hansen
2026-05-11 15:01 ` Jhonata Poma-Hansen
2026-09-23 19:44 ` [PATCH v3] " Savvas Etairidis
2026-09-24 6:36 ` [bitbake-devel] " Antonin Godard
2026-09-24 20:45 ` [PATCH v4] fetch/local: " Savvas Etairidis
2026-09-28 9:30 ` Paul Barker
2026-09-28 20:59 ` [PATCH v5] " Savvas Etairidis
2026-09-28 21:02 ` Sawas Etairidis
2026-09-29 8:22 ` Jhonata Poma-Hansen
2026-09-30 10:51 ` Paul Barker
2026-09-30 21:21 ` [PATCH v6] " Savvas Etairidis
2026-10-01 8:32 ` Paul Barker
2026-10-01 20:53 ` Savvas Etairidis [this message]
2026-10-02 8:50 ` [PATCH v7] " Paul Barker
2026-10-02 20:48 ` Sawas Etairidis
2026-10-02 20:50 ` [PATCH v8] " Savvas Etairidis
2026-10-05 14:36 ` Paul Barker
2026-10-06 16:40 ` Sawas Etairidis
2026-11-07 20:17 ` Savvas Etairidis
2026-11-07 20:23 ` [PATCH v9] " Savvas Etairidis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261001205344.99638-1-setairidis@gmail.com \
--to=setairidis@gmail.com \
--cc=bitbake-devel@lists.openembedded.org \
--cc=jhonata.poma@gmail.com \
--cc=paul@pbarker.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox