Re: [bitbake-devel] [RFC PATCH] fetch: `touch` mirror tarball stamp files on download

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2026-09-02 at 05:43 -0700, Philip Lorenz via lists.openembedded.org wrote:
> On Wed, Sep 2, 2026 at 11:06 AM, Richard Purdie wrote:
> > On Wed, 2026-09-02 at 10:32 +0200, Philip Lorenz via lists.openembedded.org wrote:
> > > The mirror tarball stamp file's mtime is currently only updated on
> > > creation / initial download. This makes it difficult to detect whether a
> > > mirror tarball is still relevant in a given build.
> > > 
> > > Improve this by also `touch`ing the mirror tarball stamp files alongside
> > > the sources primary stamp file.
> > This gets a bit tricky as we don't touch every download file at every
> > download, we only touch the end stamp file.
> > 
> > This would mean we touch mirror files and stamp files but not main
> > download files and I think brings things into more disparity?
> This patch extends bitbake to touch the mirror tarball's `.done` file (not the 
> mirror tarball itself) to bring it inline with regular stamp file handling.
> > I can also think of cases where updating the timestamp may cause things
> > to re-download the tarballs too since you can't know the content in
> > advance with many of them. This is perhaps the stronger reason not to
> > do this.
> I'm not completely familiar with all of the fetcher details (and also couldn't 
> find anything obvious) but in which cases would changing the mtime lead to a 
> redownload? I'm not sure if this is still an issue given that this only updates 
> the `.done` files of the mirror tarballs though.
>  
> The rationale behind this change is that we'd like to clean up our download caches 
> and only keep the inputs relevant to the latest build configuration around. Using 
> `.done` files (after executing a `bitbake --runall=fetch`) works for this as long 
> as the mirror tarballs are freshly created. On consecutive runs their `.done` file 
> is not touched and they are therefore deemed out of date.

I hadn't fully taken in the fact it was just the done stamps, sorry.
This does still leave me with questions though.

For example we have cases where it wouldn't try and download mirror
tarballs. If the main source repo is up to date or even just present,
it could fetch to there and the mirror tarballs wouldn't be touched.
They could even be out of date compared to the main source repo unless
the tarball generation options are set.

So in the context of having mirror tarball generation set, the patch
could make sense but probably not outside of that.

I have thoughts about changing the way mirror tarballs work in the next
release anyway so I'm still leaning towards not adding more complexity
right now...

Cheers,

Richard