Re: [bitbake-devel] [PATCH] toaster: Add recipe variables view

[Paolo Wattebled <paolo.wattebled@savoirfairelinux.com>]

Hi Richard,

> For example if the core metadata has:
>
> X = "${Y}"
> Y = "Z"
>
> then the recipe sets Y = "A", the variable history for X doesn't change
> but the value does.

You are right. My filtering happened before expansion and relied on
`VariableHistory`, which records direct operations but not changes caused by
references to other variables. It could therefore omit `X`, even though its
effective value had changed.

I removed this history-based filter. The implementation now iterates the
parsed recipe datastore and stores the expanded value of each included
variable. I added your example as a regression test, and the snapshot contains
both `X = "A"` and `Y = "A"`.

Variables marked as functions and explicit override keys are not stored.
Active overrides are represented under their logical variable names, and
non-function flags are stored as `VAR[flag]` entries. Values detected as
involving inline Python are also omitted to avoid expansion side effects.

On an `imx-image-core` build, Toaster persisted 598 recipe snapshots with an
average of 1663 entries, a minimum of 1627 and a maximum of 1944. The compressed
payload was about 86 MB in total, or 145 kB per recipe on average.

If you agree with the current scope and semantics, I will generate and send a
v3 of the series.

Thanks for pointing this out.

Cheers,
Paolo

----- Original Message -----
From: "Richard Purdie" <richard.purdie@linuxfoundation.org>
To: "Paolo Wattebled" <paolo.wattebled@savoirfairelinux.com>
Cc: "bitbake-devel" <bitbake-devel@lists.openembedded.org>
Sent: Monday, August 17, 2026 5:08:38 PM
Subject: Re: [bitbake-devel] [PATCH] toaster: Add recipe variables view

On Mon, 2026-08-17 at 15:48 -0400, Paolo Wattebled wrote:
> > The cache you've added is per parsed recipe, not per recipe used in the
> > build. Was the patch AI assisted?
> 
> The new `recipe_variables` field is collected in the recipe parse cache for
> every parsed recipe. `buildDependTree()` copies cache fields into the
> dependency graph only for recipes in the runqueue, and Toaster persists only
> those entries. Thus the cache is per parsed recipe, while persisted Toaster
> rows are runqueue-selected.
> 
> Yes, the patch was AI-assisted, as disclosed in the commit messages. I used
> AI tools to assist with code generation, but I reviewed and tested the
> changes, and I take responsibility for the design and the submitted commits.

Sorry, I did somehow miss the tag in the commit. Nonetheless, I am
worried that the changes here aren't quite right.

> > 33 differences per recipe seems very low as well.
> > 
> > bitbake -e bash | grep -v ^# > a
> > bitbake -e quilt | grep -v ^# > b
> > diff -u a b | grep ^-[A-Z] | grep -v : | wc
> > 
> > So I've filtered only variables not functions, nothing with an override
> > and I still get 132 differences.
> 
> > I think you're missing a lot of values.
> 
> The current code records only variables whose BitBake `VariableHistory` has an
> operation from the recipe, a matching `.bbappend`, or a recipe-owned `.inc`.
> It excludes class- and configuration-only values, functions, inactive
> overrides, and values that fail expansion.
> 
> Therefore the count reflects this source filter, not the complete effective
> recipe datastore. I agree that describing the result as a resolved recipe
> variable snapshot was misleading.
> 
> This raises a question about the intended scope of the feature. In your view,
> which variables should Toaster expose here? Should it provide the complete
> effective recipe datastore, including values inherited from configuration and
> classes and values selected through overrides, or should it expose a clearly
> defined subset? If you expect a subset, which inclusion or exclusion criteria
> would you consider useful and correct?

I'm not sure what the right answer is, but I don't think your current
criteria works as you think it does.

> > Ok, how does it use variable history for selection?
> 
> Variable history is used only as a provenance filter: it selects variables
> touched by recipe metadata. The stored value is then obtained separately from
> the parsed recipe datastore using `metadata.getVar(..., True)`. The history
> itself is not stored. The compressed snapshot is carried through the dependency
> graph for runqueue recipes and then stored in Toaster's `RecipeVariable` table.

For example if the core metadata has:

X = "${Y}"
Y = "Z"

then the recipe sets Y = "A", the variable history for X doesn't change
but the value does. That is a simple example, there are much more
complex ones.

Cheers,

Richard