Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[christian.leeb]

[-- Attachment #1: Type: text/plain, Size: 1148 bytes --]

Hi,

I did a bitbake virtual/kernel --runall=fetch, with a DL_DIR containing kernel git repo. This git repo does not contain the SRCREV 7818dfe57e requested.

Log suggests the fetch ran fine with the 7818dfe57e
recipe linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10: task do_fetch: Succeeded

At this point I would have expected the fetch found the repo in DL_DIR but the requested commit is t's not up-to-date hence it would reach out to the network.

Later I call bitbake again to build the kernel and I'm getting this error.

ERROR: linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10 do_unpack: Bitbake Fetcher Error: UnpackError('No up to date source found: clone directory not available or not up to date: /home/vsts-agent/azagents/build-1/_work/_temp/localdl//git2/dev.azure.com.PG-CT.PGLinux._git.pg-linux; shallow clone not enabled', 'git://dev.azure.com/PG-CT/PGLinux/_git/pg-linux;protocol=https;branch=pg-am57xx-ti-rt-linux-5.4.106')

This suggests that fetch was happy finding the repo in DL_DIR but did not care that the requested commit was missing.
Is this the intended behavior?

BR, Chris

[-- Attachment #2: Type: text/html, Size: 3159 bytes --]

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Gyorgy Sarvari]

On 12/4/25 14:56, christian.leeb via lists.yoctoproject.org wrote:
> Hi,
>
> I did a bitbake virtual/kernel --runall=fetch, with a DL_DIR
> containing kernel git repo. This git repo does not contain the SRCREV
> 7818dfe57e requested.
>
> Log suggests the fetch ran fine with the 7818dfe57e
> recipe linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10:
> task do_fetch: Succeeded
>  
> At this point I would have expected the fetch found the repo in DL_DIR
> but the requested commit is t's not up-to-date hence it would reach
> out to the network.
>  
> Later I call bitbake again to build the kernel and I'm getting this error.
>
> ERROR: linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10
> do_unpack: Bitbake Fetcher Error: UnpackError('No up to date source
> found: clone directory not available or not up to date:
> /home/vsts-agent/azagents/build-1/_work/_temp/localdl//git2/dev.azure.com.PG-CT.PGLinux._git.pg-linux;
> shallow clone not enabled',
> 'git://dev.azure.com/PG-CT/PGLinux/_git/pg-linux;protocol=https;branch=pg-am57xx-ti-rt-linux-5.4.106')
>
> This suggests that fetch was happy finding the repo in DL_DIR but did
> not care that the requested commit was missing.
> Is this the intended behavior?
>

Generally speaking this does not sound intentional - Bitbake has checks
for this, it verifies during do_fetch explicitly if the requested
revision is already present in the local clone, or it needs to be updated.

Which version of Bitbake/Yocto are you using? Also, how do you specify
the revision in the recipe? (revision, tagname in SRCREV, something else?) 
A minimal reproduction could be also useful, if available.

> BR, Chris
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#66088): https://lists.yoctoproject.org/g/yocto/message/66088
> Mute This Topic: https://lists.yoctoproject.org/mt/116612004/6084445
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [skandigraun@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[christian.leeb]

[-- Attachment #1: Type: text/plain, Size: 1796 bytes --]

I'm using Yocto 4.0, kirkstone.

We set
URI is git://dev.azure.com/PG-CT/PGLinux/_git/pg-linux;protocol=https;branch=pg-am57xx-ti-rt-linux-5.4.106
SRCREV="7818dfe57e1d1e2d2eaf039d1231560611811d67"
BRANCH="pg-am57xx-ti-rt-linux-5.4.106"

The DL_DIR contains above repo (git2/dev.azure.com.PG-CT.PGLinux._git.pg-linux) and above branch but just not the latest commit of this branch.
Also, between
recipe linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10: task do_fetch: Started
and
recipe linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10: task do_fetch: Succeeded
there is only a few seconds. Another proof IMHO that there was no network access.

Fetch works properly if the repo is not present at all in the DL_DIR.

I also check if shallow clone is set: seems that this is not the case unless there is some other setting to consider.

bitbake virtual/kernel -e | grep ^BB_GIT_SHALLOW
BB_GIT_SHALLOW:pn-binutils="1"
BB_GIT_SHALLOW:pn-binutils-cross-arm="1"
BB_GIT_SHALLOW:pn-binutils-cross-canadian-arm="1"
BB_GIT_SHALLOW:pn-binutils-cross-testsuite="1"
BB_GIT_SHALLOW:pn-binutils-crosssdk-x86_64-spark-linux="1"
BB_GIT_SHALLOW:pn-binutils-native="1"
BB_GIT_SHALLOW:pn-glibc="1"

In addition I got this info from the internet:
Why do_fetch didn’t update the repo:
BitBake’s fetcher logic:
If the repo exists in DL_DIR and the SRCREV is marked as available (based on metadata), it skips network fetch.
It doesn’t *verify the actual commit presence during do_fetch* —that check happens in do_unpack.
So the 2-second fetch means it reused the cached repo without pulling new commits.

I wonder what it means " SRCREV is marked as available (based on metadata) ". It's obviously not there.

Thanks a lot for your support.

Chris

[-- Attachment #2: Type: text/html, Size: 7914 bytes --]

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Gyorgy Sarvari]

I have been thinking about this, and also tried to reproduce it,
unsuccessfully...

If fetch is deemed successful, it is stored in the shared state cache,
and then it is not repeated until the relevant signature changes - I
think this is the metadata reference you found.
However as long as the download folder's content is only modified by one
project only, it supposed keep track of it fine.

One question about the DL_DIR: is it possible that it is shared between
multiple projects, while keeping separate shared state cache? Or that
the DL_DIR's content modified manually?
The shared state cache supposed to keep track of what is downloaded by
bitbake and what is not, but if the DL_DIR's content is changed without
bitbake (or by a separate bitbake), that can break the shared state, and
cause such issues.

On 12/4/25 22:41, christian.leeb via Lists.Yoctoproject.Org wrote:
> I'm using Yocto 4.0, kirkstone.
>  
> We set
> URI is
> git://dev.azure.com/PG-CT/PGLinux/_git/pg-linux;protocol=https;branch=pg-am57xx-ti-rt-linux-5.4.106
> SRCREV="7818dfe57e1d1e2d2eaf039d1231560611811d67"
> BRANCH="pg-am57xx-ti-rt-linux-5.4.106"
>
> The DL_DIR contains above repo
> (git2/dev.azure.com.PG-CT.PGLinux._git.pg-linux) and above branch but
> just not the latest commit of this branch.
> Also, between 
> recipe linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10:
> task do_fetch: Started
> and
> recipe linux-ti-staging-rt-5.4.106+gitAUTOINC+7818dfe57e-r7a.spark10:
> task do_fetch: Succeeded
> there is only a few seconds. Another proof IMHO that there was no
> network access.
>
> Fetch works properly if the repo is not present at all in the DL_DIR.
>
> I also check if shallow clone is set: seems that this is not the case
> unless there is some other setting to consider.
>
> bitbake virtual/kernel -e | grep ^BB_GIT_SHALLOW
> BB_GIT_SHALLOW:pn-binutils="1"
> BB_GIT_SHALLOW:pn-binutils-cross-arm="1"
> BB_GIT_SHALLOW:pn-binutils-cross-canadian-arm="1"
> BB_GIT_SHALLOW:pn-binutils-cross-testsuite="1"
> BB_GIT_SHALLOW:pn-binutils-crosssdk-x86_64-spark-linux="1"
> BB_GIT_SHALLOW:pn-binutils-native="1"
> BB_GIT_SHALLOW:pn-glibc="1"
>
> In addition I got this info from the internet:
> /Why do_fetch didn’t update the repo:/
> /BitBake’s fetcher logic:/
> /If the repo exists in DL_DIR and the SRCREV is marked as available
> (based on metadata), it skips network fetch./
> /It doesn’t *verify the actual commit presence during do_fetch*—that
> check happens in do_unpack./
> /So the 2-second fetch means it reused the cached repo without pulling
> new commits./
>
> I wonder what it means "/SRCREV is marked as available (based on
> metadata)/". It's obviously not there.
>
> Thanks a lot for your support.
>
> Chris
>

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[christian.leeb]

[-- Attachment #1: Type: text/plain, Size: 1005 bytes --]

Thanks a lot, Gyorgy, for your feedback. Maybe a bit of background what we are trying to do.

We have the following situation - we call it "top-up":
We start bitbake fetch as described with DL_DIR containing the sources of a previous release.
The state cache is empty.
The idea is that bitbake fetch only re-fetches repos which are not up-to-date.

In a regular release, we always fetch all sources anew from upstream (debatable if this is the right approach). At the end of the release we archive all sources of the dl folder. We do not archive the sstate cache.

For reproducing a release we take the archive of the release and rebuild without network (BB_NONETWORK)

The motivation for "top-up" came from last week's outage of cloudflare: we urgently needed to issue a release, but one upstream server was down. So we used dl archive of previous release, removed the "BB_NONETWORK and hoped it would fetch the missing stuff (in my example some commits of the kernel) and then we can build.

[-- Attachment #2: Type: text/html, Size: 1141 bytes --]

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Alexander Kanavin]

On Fri, 5 Dec 2025 at 17:20, christian.leeb via lists.yoctoproject.org
<christian.leeb=hitachienergy.com@lists.yoctoproject.org> wrote:
>
> Thanks a lot, Gyorgy, for your feedback. Maybe a bit of background what we are trying to do.
>
> We have the following situation - we call it "top-up":
> We start bitbake fetch as described with DL_DIR containing the sources of a previous release.
> The state cache is empty.
> The idea is that bitbake fetch only re-fetches repos which are not up-to-date.
>
> In a regular release, we always fetch all sources anew from upstream (debatable if this is the right approach). At the end of the release we archive all sources of the dl folder. We do not archive the sstate cache.
>
> For reproducing a release we take the archive of the release and rebuild without network (BB_NONETWORK)
>
> The motivation for "top-up" came from last week's outage of cloudflare: we urgently needed to issue a release, but one upstream server was down. So we used dl archive of previous release, removed the "BB_NONETWORK and hoped it would fetch the missing stuff (in my example some commits of the kernel) and then we can build.

Keeping DL_DIR between builds to enable incremental fetches is a
standard practice used by everyone, and it works, but you too need to
do it regularly to ensure there's nothing strange in your
configuration that gets in the way.

In your builds you either throw away DL_DIR, or you use it only in
offline builds, so incremental fetches are never happening. It would
perhaps help to look at log.do_fetch from that task that was supposed
to do an incremental fetch but didn't. We'd at least see what it tried
to execute to bring in the latest commits.

Otherwise you can try with a test recipe and a test git repo: make a
fetch/unpack, add commits, change SRCREV, make another fetch. It will
work. Or if it does not, we then have a way to reproduce the issue and
look into it.

Alex

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Alexander Kanavin]

On Fri, 5 Dec 2025 at 16:46, Gyorgy Sarvari via lists.yoctoproject.org
<skandigraun=gmail.com@lists.yoctoproject.org> wrote:
> I have been thinking about this, and also tried to reproduce it,
> unsuccessfully...
>
> If fetch is deemed successful, it is stored in the shared state cache,
> and then it is not repeated until the relevant signature changes - I
> think this is the metadata reference you found.
> However as long as the download folder's content is only modified by one
> project only, it supposed keep track of it fine.
>
> One question about the DL_DIR: is it possible that it is shared between
> multiple projects, while keeping separate shared state cache? Or that
> the DL_DIR's content modified manually?
> The shared state cache supposed to keep track of what is downloaded by
> bitbake and what is not, but if the DL_DIR's content is changed without
> bitbake (or by a separate bitbake), that can break the shared state, and
> cause such issues.

This is not correct. Shared state does not track what is downloaded.
It is also fine to share DL_DIR between multiple builds and projects.

Alex

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Gyorgy Sarvari]

On 12/5/25 18:24, Alexander Kanavin wrote:
> On Fri, 5 Dec 2025 at 16:46, Gyorgy Sarvari via lists.yoctoproject.org
> <skandigraun=gmail.com@lists.yoctoproject.org> wrote:
>> I have been thinking about this, and also tried to reproduce it,
>> unsuccessfully...
>>
>> If fetch is deemed successful, it is stored in the shared state cache,
>> and then it is not repeated until the relevant signature changes - I
>> think this is the metadata reference you found.
>> However as long as the download folder's content is only modified by one
>> project only, it supposed keep track of it fine.
>>
>> One question about the DL_DIR: is it possible that it is shared between
>> multiple projects, while keeping separate shared state cache? Or that
>> the DL_DIR's content modified manually?
>> The shared state cache supposed to keep track of what is downloaded by
>> bitbake and what is not, but if the DL_DIR's content is changed without
>> bitbake (or by a separate bitbake), that can break the shared state, and
>> cause such issues.
> This is not correct. Shared state does not track what is downloaded.

It depends on how you interpret it. Sstate does track the do_fetch task,
and does know if it needs to re-executed or not.

> It is also fine to share DL_DIR between multiple builds and projects.

I did not write that it is not fine to share DL_DIR, but but you need to
know some caveats. If you run bitbake -c cleanall in one project, you
are up for a good time in the other one, if they use the same recipe.

>
> Alex

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Alexander Kanavin]

On Fri, 5 Dec 2025 at 19:12, Gyorgy Sarvari <skandigraun@gmail.com> wrote:
> >> If fetch is deemed successful, it is stored in the shared state cache,
> >> and then it is not repeated until the relevant signature changes - I
> >> think this is the metadata reference you found.
> >> However as long as the download folder's content is only modified by one
> >> project only, it supposed keep track of it fine.
> >>
> >> One question about the DL_DIR: is it possible that it is shared between
> >> multiple projects, while keeping separate shared state cache? Or that
> >> the DL_DIR's content modified manually?
> >> The shared state cache supposed to keep track of what is downloaded by
> >> bitbake and what is not, but if the DL_DIR's content is changed without
> >> bitbake (or by a separate bitbake), that can break the shared state, and
> >> cause such issues.
> > This is not correct. Shared state does not track what is downloaded.
> It depends on how you interpret it. Sstate does track the do_fetch task,
> and does know if it needs to re-executed or not.

The process by which bitbake determines which tasks need to be
executed, and in which order, is well defined, and it is not like
that.

This is how it is:
1. Bitbake unrolls the target specified on a command line into a
directed graph, where nodes are tasks and edges are task dependencies.
Each task is given a signature with a hash.
2. For each task bitbake checks if there's a stamp in the build
directory with a matching hash (meaning the build directory already
has the outcome of the task), and if so, the task, and everything it
depends on, is pruned from the graph.
3. For each task that is sstate-enabled (neither fetch nor unpack
tasks are in that set), bitbake checks if sstate contains an object
matching the task signature. If so, everything that the task depends
on is pruned from the graph, and the task is replaced with its
setscene variant that unpacks the sstate.
4. Remaining tasks are executed in order that respects their dependencies.

Sstate cache (or code that manages it) does not track tasks, it only
stores and retrieves sstate objects, and it does not know anything
about what tasks in a particular build need to be executed.

> > It is also fine to share DL_DIR between multiple builds and projects.
>
> I did not write that it is not fine to share DL_DIR, but but you need to
> know some caveats. If you run bitbake -c cleanall in one project, you
> are up for a good time in the other one, if they use the same recipe.

That is fine to do as well, as long as no other builds are running at
the same time as cleanall. The other project either does not need to
run fetch/unpack and won't notice sources are absent, or it will run
fetch, fetch will notice the sources are absent in DL_DIR, and then it
will simply re-fetch them.

Alex

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Gyorgy Sarvari]

On 12/8/25 11:31, Alexander Kanavin wrote:
> On Fri, 5 Dec 2025 at 19:12, Gyorgy Sarvari <skandigraun@gmail.com> wrote:
>>>> If fetch is deemed successful, it is stored in the shared state cache,
>>>> and then it is not repeated until the relevant signature changes - I
>>>> think this is the metadata reference you found.
>>>> However as long as the download folder's content is only modified by one
>>>> project only, it supposed keep track of it fine.
>>>>
>>>> One question about the DL_DIR: is it possible that it is shared between
>>>> multiple projects, while keeping separate shared state cache? Or that
>>>> the DL_DIR's content modified manually?
>>>> The shared state cache supposed to keep track of what is downloaded by
>>>> bitbake and what is not, but if the DL_DIR's content is changed without
>>>> bitbake (or by a separate bitbake), that can break the shared state, and
>>>> cause such issues.
>>> This is not correct. Shared state does not track what is downloaded.
>> It depends on how you interpret it. Sstate does track the do_fetch task,
>> and does know if it needs to re-executed or not.
> The process by which bitbake determines which tasks need to be
> executed, and in which order, is well defined, and it is not like
> that.
>
> This is how it is:
> 1. Bitbake unrolls the target specified on a command line into a
> directed graph, where nodes are tasks and edges are task dependencies.
> Each task is given a signature with a hash.
> 2. For each task bitbake checks if there's a stamp in the build
> directory with a matching hash (meaning the build directory already
> has the outcome of the task), and if so, the task, and everything it
> depends on, is pruned from the graph.
> 3. For each task that is sstate-enabled (neither fetch nor unpack
> tasks are in that set), bitbake checks if sstate contains an object
> matching the task signature. If so, everything that the task depends
> on is pruned from the graph, and the task is replaced with its
> setscene variant that unpacks the sstate.
> 4. Remaining tasks are executed in order that respects their dependencies.
>
> Sstate cache (or code that manages it) does not track tasks, it only
> stores and retrieves sstate objects, and it does not know anything
> about what tasks in a particular build need to be executed.
>
>>> It is also fine to share DL_DIR between multiple builds and projects.
>> I did not write that it is not fine to share DL_DIR, but but you need to
>> know some caveats. If you run bitbake -c cleanall in one project, you
>> are up for a good time in the other one, if they use the same recipe.
> That is fine to do as well, as long as no other builds are running at
> the same time as cleanall. The other project either does not need to
> run fetch/unpack and won't notice sources are absent, or it will run
> fetch, fetch will notice the sources are absent in DL_DIR, and then it
> will simply re-fetch them.

1. Set up a new project project1, with a custom dl_dir
2. Set up a new project project2, with the same custom dl_dir
3. Run bitbake -c fetch $RECIPE in both project1
4. Run bitbake -c fetch $RECIPE in both project2
5. Run bitbake -c cleanall $RECIPE in project1
6. Run bitbake $RECIPE in project2.

Once again - using shared DL_DIR is fine usually (and I never even
implied the opposite of it), but there are gotchas.

On a related note: this is not an order of course, but if you could not
reply, that would be great. I will also go out of my way to avoid direct
communication with you now, and in the future.

> Alex

Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[christian.leeb]

[-- Attachment #1: Type: text/plain, Size: 154 bytes --]

Deeply sorry - was a logical error on my side - a bitbake var was was wrongly set in my twisted pipeline setup.
Thanks a log for your support.

Chris

[-- Attachment #2: Type: text/html, Size: 204 bytes --]