Re: [docs] [PATCH 00/10] docs: highlight BitBake snippets with the bitbake language

[Trevor Woerner <twoerner@gmail.com>]

On Mon 2026-08-31 @ 11:01:44 AM, Antonin Godard wrote:
> Hi Quentin, Trevor,
> 
> After having thought about this a bit more, I think there are pros and cons to
> both approaches, but I'm leaning towards Quentin's approach. The other approach
> may feel "safe", and doesn't leave room to errors, but I'm afraid that 1. most
> contributors will forget about it and 2. it might make the process more painful
> for them.
> 
> So here what we should do:
> 
> - set `hightlight_language = "bitbake"` in conf.py.
> 
> - remove file-wide lexer enforcing (.. highlight:: directive at the top of the
>   file) (in a separate patch)
> 
> - use the appropriate lexer for each code-block that is _not_ bitbake code.
> 
> - fix any parsing error from the bitbake lexer (I did have some when trying it).
>   If this happens, use "none" and *add a identifiable comment* above it to
>   explain that there's an issue with the lexer. This way we can track them and
>   fix them when Pygments gets an update.

The pygments releases occur on a rather slow timeline, I predict the next
release will probably be in Dec if not Jan 2027, if history is any
indicator.

There's a problem (Quentin mentioned it in one of his replies): users
don't use the tarball that the AB uses (in general, I assume) and
versions are not pinned. So users are free to use whatever is on their
system but hopefully have created a venv. But even if they're using a
venv there's no guarantee that they're updating their tools regularly.

So we're left with the following situation:
- the AB has to wait until the tarball is updated
- users might be using older versions of pygments

And then on top of that you layer on the situation of trying to generate
docs for older releases not to mention backports.

However, I can add a shim so that everything works out of the box
today. The shim can be smart enough to examine the bitbake support
independently (at runtime) and only load itself when it is needed
(either an older version of pygments that has no support, or the 2.21.0
version that needs additional support). If/when pygments is updated
(after the next release and either because the user has updated their
tools or the tarball has been updated) the shim will not load itself.

Carrying a bitbake language shim in the docs repository itself:
- the AB doesn't have to wait for an update, bitbake works today on all
  valid snippets
- backports can start working today too, since tooling doesn't have to
  be updated, the shim knows how to highlight bitbake independent of
  tools or versions
- we don't have to wait for the next release (5-6 months) for all
  the bitbake snippets to parse correctly

The nice thing about the shim (the way I've designed it) is it is
dynamic. It will look for any bitbake support in the currently used
tools. After running a tiny bit of testing it will load itself only:
- if there is no bitbake support
- if the bitbake support is incomplete
Otherwise it won't load and won't interfere.

Even once the next release of pygments occurs and full bitbake support
exists, it will still be a good idea to carry the shim so that older
docs, users with older tools, and the AB doesn't have to wait for a new
tarball to get full bitbake highlighting.

Also, this way we don't have to skip the non-working snippets today
with a comments, all snippets will highlight today, no need to go back
and fix things up in 6 months if someone remembers, etc. Full bitbake
highlighting for everyone under any circumstance starting today and
available forever across all versions of docs and tools.

> 
> - document this way of proceeding in standards.md.
> 
> If you think of anything else we should do here, please share your thoughts :)
> 
> Trevor, if you want to send another version of your series, please rebase on
> *master-next* as it contains cleanups to the conf.py file at the moment (+ some
> possible additional code-blocks).
> 
> Thanks both for sharing your thoughts on this!
> 
> Antonin