Re: [yocto] Bitbake fetch does not re-fetch from network if gitrepo is found in DL_DIR

[Gyorgy Sarvari <skandigraun@gmail.com>]

On 12/8/25 11:31, Alexander Kanavin wrote:
> On Fri, 5 Dec 2025 at 19:12, Gyorgy Sarvari <skandigraun@gmail.com> wrote:
>>>> If fetch is deemed successful, it is stored in the shared state cache,
>>>> and then it is not repeated until the relevant signature changes - I
>>>> think this is the metadata reference you found.
>>>> However as long as the download folder's content is only modified by one
>>>> project only, it supposed keep track of it fine.
>>>>
>>>> One question about the DL_DIR: is it possible that it is shared between
>>>> multiple projects, while keeping separate shared state cache? Or that
>>>> the DL_DIR's content modified manually?
>>>> The shared state cache supposed to keep track of what is downloaded by
>>>> bitbake and what is not, but if the DL_DIR's content is changed without
>>>> bitbake (or by a separate bitbake), that can break the shared state, and
>>>> cause such issues.
>>> This is not correct. Shared state does not track what is downloaded.
>> It depends on how you interpret it. Sstate does track the do_fetch task,
>> and does know if it needs to re-executed or not.
> The process by which bitbake determines which tasks need to be
> executed, and in which order, is well defined, and it is not like
> that.
>
> This is how it is:
> 1. Bitbake unrolls the target specified on a command line into a
> directed graph, where nodes are tasks and edges are task dependencies.
> Each task is given a signature with a hash.
> 2. For each task bitbake checks if there's a stamp in the build
> directory with a matching hash (meaning the build directory already
> has the outcome of the task), and if so, the task, and everything it
> depends on, is pruned from the graph.
> 3. For each task that is sstate-enabled (neither fetch nor unpack
> tasks are in that set), bitbake checks if sstate contains an object
> matching the task signature. If so, everything that the task depends
> on is pruned from the graph, and the task is replaced with its
> setscene variant that unpacks the sstate.
> 4. Remaining tasks are executed in order that respects their dependencies.
>
> Sstate cache (or code that manages it) does not track tasks, it only
> stores and retrieves sstate objects, and it does not know anything
> about what tasks in a particular build need to be executed.
>
>>> It is also fine to share DL_DIR between multiple builds and projects.
>> I did not write that it is not fine to share DL_DIR, but but you need to
>> know some caveats. If you run bitbake -c cleanall in one project, you
>> are up for a good time in the other one, if they use the same recipe.
> That is fine to do as well, as long as no other builds are running at
> the same time as cleanall. The other project either does not need to
> run fetch/unpack and won't notice sources are absent, or it will run
> fetch, fetch will notice the sources are absent in DL_DIR, and then it
> will simply re-fetch them.

1. Set up a new project project1, with a custom dl_dir
2. Set up a new project project2, with the same custom dl_dir
3. Run bitbake -c fetch $RECIPE in both project1
4. Run bitbake -c fetch $RECIPE in both project2
5. Run bitbake -c cleanall $RECIPE in project1
6. Run bitbake $RECIPE in project2.

Once again - using shared DL_DIR is fine usually (and I never even
implied the opposite of it), but there are gotchas.

On a related note: this is not an order of course, but if you could not
reply, that would be great. I will also go out of my way to avoid direct
communication with you now, and in the future.

> Alex