Re: [bitbake-devel] [PATCH] toaster: Add recipe variables view

[Paolo Wattebled <paolo.wattebled@savoirfairelinux.com>]

Hi Richard,

Thanks for raising this. I ran a quick comparison with and without the change.
Toaster startup and recipe parsing were effectively unchanged (2m46.554s vs
2m46.835s). A cached imx-image-core build took 4m23.648s versus 4m07.387s,
about 16 seconds slower. After vacuuming, both SQLite databases rounded to 74
MiB, including 598 per-recipe variable snapshots. I understand that some of
this data is stored twice. However, I only store variables changed by recipes
or bbappends, and I compress them before saving them. In my test, this did not
noticeably increase the database size.

Our intended use of Toaster is as the backend for an MCP server for AI agents.
Rather than having agents repeatedly parse and search recipe files or invoke
bitbake-getvar, the server would expose resolved, build-specific metadata
through a stable API. The MCP layer is still under development, but per-recipe
variable values are one of its planned endpoints. We chose Toaster because it
already models builds, recipes, packages and variables; extending it avoids
creating a separate metadata store. It will run headlessly in this use case,
with the MCP server translating between Toaster’s API and the agent.

Regards,
Paolo Wattebled


----- Original Message -----
From: "Richard Purdie" <richard.purdie@linuxfoundation.org>
To: "Paolo Wattebled" <paolo.wattebled@savoirfairelinux.com>, bitbake-devel@lists.openembedded.org
Sent: Wednesday, August 12, 2026 4:34:24 PM
Subject: Re: [bitbake-devel] [PATCH] toaster: Add recipe variables view

On Wed, 2026-08-12 at 11:04 -0400, Paolo Wattebled via lists.openembedded.org wrote:
> Toaster records global build variables but does not expose the final
> values associated with individual recipes.
> 
> Collect variables defined or modified by recipes and their bbappends,
> resolve active overrides and expansions, and store one compressed JSON
> snapshot in the database for each build and recipe. Keep the variable
> count separately to avoid decompressing snapshots in recipe-list
> queries.
> 
> Expose the values through a searchable and paginated Variables tab and
> show their counts in the built-recipes table and recipe package view.
> Preserve compatibility with older dependency payloads and invalidate
> stale BitBake caches after extending the cached recipe information.
> 
> Tests cover variable collection, cache transport, database persistence,
> backward compatibility, searching, pagination, escaping, and counts.
> 
> AI-Generated: Uses GitHub Copilot and OpenCode with GPT-5.6 Sol
> Signed-off-by: Paolo Wattebled <paolo.wattebled@savoirfairelinux.com>
> ---
>  bin/bitbake-selftest                          |   1 +
>  lib/bb/cache.py                               |   2 +-
>  lib/bb/cache_extra.py                         |  66 +++++++-
>  lib/bb/cooker.py                              |   6 +-
>  lib/bb/tests/cache_extra.py                   | 141 ++++++++++++++++
>  lib/bb/ui/buildinfohelper.py                  |  21 ++-
>  .../orm/migrations/0022_recipevariable.py     |  27 +++
>  lib/toaster/orm/models.py                     |  10 ++
>  lib/toaster/tests/db/test_db.py               | 101 ++++++++++++
>  lib/toaster/tests/views/test_views.py         | 155 +++++++++++++++++-
>  lib/toaster/toastergui/buildtables.py         |  11 +-
>  .../templates/detail_pagination_bottom.html   |   4 +-
>  .../templates/detail_search_header.html       |   2 +-
>  lib/toaster/toastergui/templates/recipe.html  |  32 ++++
>  .../toastergui/templates/recipe_packages.html |   7 +
>  lib/toaster/toastergui/views.py               |  68 +++++++-
>  16 files changed, 642 insertions(+), 12 deletions(-)
>  create mode 100644 lib/bb/tests/cache_extra.py
>  create mode 100644 lib/toaster/orm/migrations/0022_recipevariable.py

At this point you're basically dumping out the entire datastore for
every possible recipe. Due to the size of the data, I suspect it is
just as expensive to reparse as it is to read the stored compressed
data and it isn't really a cache any more. The presence of that data
will slow the rest of the system down.

Have you looked into the performance (in speed, memory and disk space)
implications of this?

Also, do you know people actively using toaster?

Cheers,

Richard