Re: [docs] [PATCH 00/10] docs: highlight BitBake snippets with the bitbake language

[Trevor Woerner <twoerner@gmail.com>]

On Mon 2026-08-31 @ 05:19:37 PM, Antonin Godard wrote:
> On Mon Aug 31, 2026 at 4:16 PM CEST, Trevor Woerner wrote:
> > On Mon 2026-08-31 @ 11:01:44 AM, Antonin Godard wrote:
> >> Hi Quentin, Trevor,
> >> 
> >> After having thought about this a bit more, I think there are pros and cons to
> >> both approaches, but I'm leaning towards Quentin's approach. The other approach
> >> may feel "safe", and doesn't leave room to errors, but I'm afraid that 1. most
> >> contributors will forget about it and 2. it might make the process more painful
> >> for them.
> >> 
> >> So here what we should do:
> >> 
> >> - set `hightlight_language = "bitbake"` in conf.py.
> >> 
> >> - remove file-wide lexer enforcing (.. highlight:: directive at the top of the
> >>   file) (in a separate patch)
> >> 
> >> - use the appropriate lexer for each code-block that is _not_ bitbake code.
> >> 
> >> - fix any parsing error from the bitbake lexer (I did have some when trying it).
> >>   If this happens, use "none" and *add a identifiable comment* above it to
> >>   explain that there's an issue with the lexer. This way we can track them and
> >>   fix them when Pygments gets an update.
> >
> > The pygments releases occur on a rather slow timeline, I predict the next
> > release will probably be in Dec if not Jan 2027, if history is any
> > indicator.
> >
> > There's a problem (Quentin mentioned it in one of his replies): users
> > don't use the tarball that the AB uses (in general, I assume) and
> > versions are not pinned. So users are free to use whatever is on their
> > system but hopefully have created a venv. But even if they're using a
> > venv there's no guarantee that they're updating their tools regularly.
> 
> We can pin the version required to build the docs in
> documentation/tools/host_packages_scripts/pip3_docs.sh. This will become the
> minimum version of Pygments required to build the docs.
> 
> > So we're left with the following situation:
> > - the AB has to wait until the tarball is updated
> > - users might be using older versions of pygments
> >
> > And then on top of that you layer on the situation of trying to generate
> > docs for older releases not to mention backports.
> 
> The change in conf.py will not impact older releases, so they should be safe to
> build, even if Pygments is updated with the bitbake lexer: it simply won't be
> used, because highlight_language is still "default" on these.

If we set:

	highlight_language = "bitbake"

in conf.py right now, anyone who is not running pygments 2.21.0 will get
warnings which translate to build failures via the Makefile's -W
setting.

> Note: I think this change is big enough that I don't consider it candidate for
> backport on stable branches.
> 
> > However, I can add a shim so that everything works out of the box
> > today. The shim can be smart enough to examine the bitbake support
> > independently (at runtime) and only load itself when it is needed
> > (either an older version of pygments that has no support, or the 2.21.0
> > version that needs additional support). If/when pygments is updated
> > (after the next release and either because the user has updated their
> > tools or the tarball has been updated) the shim will not load itself.
> >
> > Carrying a bitbake language shim in the docs repository itself:
> > - the AB doesn't have to wait for an update, bitbake works today on all
> >   valid snippets
> > - backports can start working today too, since tooling doesn't have to
> >   be updated, the shim knows how to highlight bitbake independent of
> >   tools or versions
> > - we don't have to wait for the next release (5-6 months) for all
> >   the bitbake snippets to parse correctly
> >
> > The nice thing about the shim (the way I've designed it) is it is
> > dynamic. It will look for any bitbake support in the currently used
> > tools. After running a tiny bit of testing it will load itself only:
> > - if there is no bitbake support
> > - if the bitbake support is incomplete
> > Otherwise it won't load and won't interfere.
> >
> > Even once the next release of pygments occurs and full bitbake support
> > exists, it will still be a good idea to carry the shim so that older
> > docs, users with older tools, and the AB doesn't have to wait for a new
> > tarball to get full bitbake highlighting.
> >
> > Also, this way we don't have to skip the non-working snippets today
> > with a comments, all snippets will highlight today, no need to go back
> > and fix things up in 6 months if someone remembers, etc. Full bitbake
> > highlighting for everyone under any circumstance starting today and
> > available forever across all versions of docs and tools.
> 
> Considering what I've said above, do you really think that's necessary? To me it
> feels like increased complexity considering we'll only add this feature on
> master, and it should be straightforward from there to fix future stable
> releases with backport patches when Pygments gets updated.

What's the complexity? The shim is written, it's done, it works
regardless of which pygments version anyone (AB, users) are using and it
doesn't install if the conditions are met. No waiting, everyone gets all
bitbake highlighted today.

If anything not adding the shim *is* where the complexity lies. Once you
accept the default conf.py patch the AB must be using the new pygments,
and users will see build errors, and report them, until we point out
they have to update their tools.

> Antonin