[PATCH] fetch2: avoid reuse download filenames

[PATCH] fetch2: avoid reuse download filenames

[Pedro Ferreira]

When fetch task runs and while running checksum validation
detects that for a source file the checksum mismatches,
instead of aborting, its allowing to move aside and download again.
This might allow users to taint the source files instead of acting
as a safe mechanism to fix some issue occurred on the download stage.

Signed-off-by: Pedro Ferreira <Pedro.MS.Ferreira@ctw.mbwgroup.com>
---
 lib/bb/fetch2/__init__.py | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index dcf7adc26..b8d8957b0 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -716,13 +716,10 @@ def verify_donestamp(ud, d, origud=None):
                 p.dump(checksums)
         return True
     except ChecksumError as e:
-        # Checksums failed to verify, trigger re-download and remove the
-        # incorrect stamp file.
-        logger.warning("Checksum mismatch for local file %s\n"
-                       "Cleaning and trying again." % ud.localpath)
-        if os.path.exists(ud.localpath):
-            rename_bad_checksum(ud, e.checksum)
-        bb.utils.remove(ud.donestamp)
+        # If there is a checksum mismatch, it is likely because the file
+        # is being tainted or some corruption is occurring when downloading.
+        # Download cache should be cleaned up before trying again.
+        bb.fatal("Checksum mismatch for local file %s\n" % ud.localpath)
     return False
 
 
-- 
2.43.0

Re: [bitbake-devel] [PATCH] fetch2: avoid reuse download filenames

[Richard Purdie]

On Mon, 2026-03-02 at 15:14 +0000, Pedro Ferreira via lists.openembedded.org wrote:
> When fetch task runs and while running checksum validation
> detects that for a source file the checksum mismatches,
> instead of aborting, its allowing to move aside and download again.
> This might allow users to taint the source files instead of acting
> as a safe mechanism to fix some issue occurred on the download stage.
> 
> Signed-off-by: Pedro Ferreira <Pedro.MS.Ferreira@ctw.mbwgroup.com>
> ---
>  lib/bb/fetch2/__init__.py | 11 ++++-------
>  1 file changed, 4 insertions(+), 7 deletions(-)

Do you have a bit more information about the issue you ran into?

This behaviour was added so that a "bad" mirror with a broken file
wouldn't stop a build when there were mirrors with the correct file
present. That is beneficial for some scenarios.

Cheers,

Richard


> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index dcf7adc26..b8d8957b0 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -716,13 +716,10 @@ def verify_donestamp(ud, d, origud=None):
>                  p.dump(checksums)
>          return True
>      except ChecksumError as e:
> -        # Checksums failed to verify, trigger re-download and remove the
> -        # incorrect stamp file.
> -        logger.warning("Checksum mismatch for local file %s\n"
> -                       "Cleaning and trying again." % ud.localpath)
> -        if os.path.exists(ud.localpath):
> -            rename_bad_checksum(ud, e.checksum)
> -        bb.utils.remove(ud.donestamp)
> +        # If there is a checksum mismatch, it is likely because the file
> +        # is being tainted or some corruption is occurring when downloading.
> +        # Download cache should be cleaned up before trying again.
> +        bb.fatal("Checksum mismatch for local file %s\n" % ud.localpath)
>      return False
>  
>  
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#19097): https://lists.openembedded.org/g/bitbake-devel/message/19097
> Mute This Topic: https://lists.openembedded.org/mt/118095154/1686473
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [richard.purdie@linuxfoundation.org]
> -=-=-=-=-=-=-=-=-=-=-=-

Re: [PATCH] fetch2: avoid reuse download filenames

[pedro.ms.ferreira]

[-- Attachment #1: Type: text/plain, Size: 1475 bytes --]

Hi Richard,

This was detected internally when users modify source files and update
the hashes inside the recipes to match. Bitbake is allowing this operations
because it identifies a mismatch and renames it to procede with the download.

I tested this locally with:

- Create a source file, for example, test_download.tar.gz
- Create a simple http server (python3 -m http.server 1212 )
- Create a small recipe, local_test.bb
------------------------------------------------------------------------
SUMMARY = "local test recipe"
DESCRIPTION = "Downloads and installs test files"
LICENSE = "CLOSED"

SRC_URI = "http://0.0.0.0:1212/test_download.tar.gz"

S = "${WORKDIR}"

SRC_URI[md5sum] = "12e597833772126eade44740570aab3c"
SRC_URI[sha256sum] = "f03e325aa451beda1b340a7ce0fcfd620b74c86e12b8156a3d4949f7de67fc87"

do_install() {
install -d ${D}/etc/test
install  ${S}/README.md -C ${D}/etc/test
}

FILES:${PN} = "/etc/test/README.md"

------------------------------------------------------------------------
- Run it: bitbake -C fetch local-test
- Modify source file
- Update the hash
- Run it: bitbake -C fetch local-test
- A warning is raised: WARNING: local-test-1.0-r0 do_fetch: Renaming /home/test/yocto-downloads/test_download.tar.gz to /home/test/yocto-downloads/test_download.tar.gz_bad-checksum_f03e325aa451beda1b340a7ce0fcfd620b74c86e12b8156a3d4949f7de67fc87
- Bitbake continues the next steps with this new tarball.

Cheers.

[-- Attachment #2: Type: text/html, Size: 2037 bytes --]

Re: [bitbake-devel] [PATCH] fetch2: avoid reuse download filenames

[Yoann Congal]

On Mon Mar 2, 2026 at 5:54 PM CET, pedro.ms.ferreira via lists.openembedded.org wrote:
> Hi Richard,
>
> This was detected internally when users modify source files and update
> the hashes inside the recipes to match. Bitbake is allowing this operations
> because it identifies a mismatch and renames it to procede with the download.
>
> I tested this locally with:
>
> - Create a source file, for example, test_download.tar.gz
> - Create a simple http server (python3 -m http.server 1212 )
> - Create a small recipe, local_test.bb
> ------------------------------------------------------------------------
> SUMMARY = "local test recipe"
> DESCRIPTION = "Downloads and installs test files"
> LICENSE = "CLOSED"
>
> SRC_URI = "http://0.0.0.0:1212/test_download.tar.gz"
>
> S = "${WORKDIR}"
>
> SRC_URI[md5sum] = "12e597833772126eade44740570aab3c"
> SRC_URI[sha256sum] = "f03e325aa451beda1b340a7ce0fcfd620b74c86e12b8156a3d4949f7de67fc87"
>
> do_install() {
> install -d ${D}/etc/test
> install  ${S}/README.md -C ${D}/etc/test
> }
>
> FILES:${PN} = "/etc/test/README.md"
>
> ------------------------------------------------------------------------
> - Run it: bitbake -C fetch local-test
> - Modify source file
> - Update the hash
> - Run it: bitbake -C fetch local-test
> - A warning is raised: WARNING: local-test-1.0-r0 do_fetch: Renaming /home/test/yocto-downloads/test_download.tar.gz to /home/test/yocto-downloads/test_download.tar.gz_bad-checksum_f03e325aa451beda1b340a7ce0fcfd620b74c86e12b8156a3d4949f7de67fc87
> - Bitbake continues the next steps with this new tarball.
>
> Cheers.

Note: This is related to 16154 – Mirroring process fails if an archive changes hash without changing name
https://bugzilla.yoctoproject.org/show_bug.cgi?id=16154
-- 
Yoann Congal
Smile ECS

Re: [PATCH] fetch2: avoid reuse download filenames

[pedro.ms.ferreira]

[-- Attachment #1: Type: text/plain, Size: 3964 bytes --]

Hi Richard,

I tested a few scenarios to be sure that this is not blocking or changing behavior apart from the one use case that im trying to fix.

Use case 1
- No local file
- Two mirrors with the source file:
- Mirror 1: 7da977765ddfe7a38e72905d6e2ae71b
- Mirror 2: 12e597833772126eade44740570aab3c
- Recipe points SRC_URI to 12e597833772126eade44740570aab3c
- PREMIRRORS appended with both http servers

bitbake -C fetch local-test - Succeeds
---------------------------------------------------------------------------------------------------
WARNING: local-test-1.0-r0 do_fetch: Checksum mismatch for local file (...) /yocto-downloads/test_download.tar.gz
Cleaning and trying again.
WARNING: local-test-1.0-r0 do_fetch: Renaming (...)/yocto-downloads/ test_download.tar.gz to (...) /yocto-downloads/ test_download.tar.gz_bad-checksum_f03e325aa451beda1b340a7ce0fcfd620b74c86e12b8156a3d4949f7de67fc87
WARNING: local-test-1.0-r0 do_fetch: Checksum failure encountered with premirror download of http://0.0.0.0:1234/ test_download.tar.gz - will attempt other sources.
---------------------------------------------------------------------------------------------------

Use case 2
- No local file
- Two mirrors with the source file:
- Mirror 1: 7da977765ddfe7a38e72905d6e2ae71b
- Mirror 2: 7da977765ddfe7a38e72905d6e2ae71b
- Recipe points SRC_URI to 12e597833772126eade44740570aab3c
- PREMIRRORS appended with both http servers

bitbake -C fetch local-test - Fails
---------------------------------------------------------------------------------------------------
ERROR: local-test-1.0-r0 do_fetch: Fetcher failure for URL: 'http://0.0.0.0:1234/test_download.tar.gz'. Checksum mismatch!
File: '(...)/yocto-downloads/test_download.tar.gz.tmp' has sha256 checksum 'f03e325aa451beda1b340a7ce0fcfd620b74c86e12b8156a3d4949f7de67fc87' when '0f27e2d871ee8ae4784e4461ea2c35190df94ec8e8b07caba80056b2520999e1' was expected
If this change is expected (e.g. you have upgraded to a new version without updating the checksums) then you can use these lines within the recipe:
SRC_URI[sha256sum] = "f03e325aa451beda1b340a7ce0fcfd620b74c86e12b8156a3d4949f7de67fc87"
Otherwise you should retry the download and/or check with upstream to determine if the file has become corrupted or otherwise unexpectedly modified.
ERROR: local-test-1.0-r0 do_fetch: Bitbake Fetcher Error: FetchError('Unable to fetch URL from any source.', 'http://0.0.0.0:1234/test_download.tar.gz')
ERROR: Logfile of failure stored in:(...)/qemux86-64/build/tmp/work/core2-64-poky-linux/local-test/1.0/temp/log.do_fetch.13826
ERROR: Task (.../local-test/local-test.bb:do_fetch) failed with exit code '1'
---------------------------------------------------------------------------------------------------

Use case 3 (Target scenario with the patch proposed)
- Local file (7da977765ddfe7a38e72905d6e2ae71b)
- Two mirrors with the source file:
- Mirror 1: 12e597833772126eade44740570aab3c
- Mirror 2: 12e597833772126eade44740570aab3c
- Recipe points SRC_URI to 12e597833772126eade44740570aab3c
- PREMIRRORS appended with both http servers

bitbake -C fetch local-test - Fails
---------------------------------------------------------------------------------------------------
ERROR: local-test-1.0-r0 do_fetch: Checksum mismatch for local file (...)/yocto-downloads/test_download.tar.gz
ERROR: Logfile of failure stored in: (...)/qemux86-64/build/tmp/work/core2-64-poky-linux/local-test/1.0/temp/log.do_fetch.28967
ERROR: Task (.../local-test/local-test.bb:do_fetch) failed with exit code '1'
---------------------------------------------------------------------------------------------------

I think this covers at least the most common cases, imho it makes sense to block any attempt of users to overwrite old source files,
and this should be user responsibility to fix the recipe.
If you have any other use case in mind please let me know.

Cheers.

[-- Attachment #2: Type: text/html, Size: 5244 bytes --]

Re: [bitbake-devel] [PATCH] fetch2: avoid reuse download filenames

[Mathieu Dubois-Briand]

On Mon Mar 2, 2026 at 4:14 PM CET, Pedro Ferreira via lists.openembedded.org wrote:
> When fetch task runs and while running checksum validation
> detects that for a source file the checksum mismatches,
> instead of aborting, its allowing to move aside and download again.
> This might allow users to taint the source files instead of acting
> as a safe mechanism to fix some issue occurred on the download stage.
>
> Signed-off-by: Pedro Ferreira <Pedro.MS.Ferreira@ctw.mbwgroup.com>
> ---

Hi Pedro,

Thanks for your patch.

I suspect this is making the bitbake-selftest fail:

ERROR: test_invalid_missing_donestamp (bb.tests.fetch.FetcherNoNetworkTest.test_invalid_missing_donestamp)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 710, in verify_donestamp
    checksums = verify_checksum(ud, d, precomputed_checksums)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/bitbake/lib/bb/fetch2/__init__.py", line 660, in verify_checksum
    raise ChecksumError("\n".join(messages), ud.url, bad_checksum)
bb.fetch2.ChecksumError: Fetcher failure for URL: 'http://invalid.yoctoproject.org/test-file.tar.gz'. Checksum mismatch!
File: '/tmp/bitbake-fetch-gc0zy9ws/download/test-file.tar.gz' has md5 checksum 'd41d8cd98f00b204e9800998ecf8427e' when '4221d002ceb5d3c9e9137e495ceaa647' was expected
File: '/tmp/bitbake-fetch-gc0zy9ws/download/test-file.tar.gz' has sha256 checksum 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' when

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3454
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3224

Can you have a look at these errors?

Thanks,
Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCH] fetch2: avoid reuse download filenames

[pedro.ms.ferreira]

[-- Attachment #1: Type: text/plain, Size: 196 bytes --]

Hi Mathieu,

Thanks fot the heads up about the unit test.

v2 created to include the unit test rework.

https://lists.openembedded.org/g/bitbake-devel/topic/patch_v2_fetch2_avoid/118153439

[-- Attachment #2: Type: text/html, Size: 291 bytes --]