Re: [docs] [PATCH 00/10] docs: highlight BitBake snippets with the bitbake language

["Antonin Godard" <antonin.godard@bootlin.com>]

On Mon Aug 31, 2026 at 5:55 PM CEST, Trevor Woerner wrote:
> On Mon 2026-08-31 @ 05:19:37 PM, Antonin Godard wrote:
>> On Mon Aug 31, 2026 at 4:16 PM CEST, Trevor Woerner wrote:
>> > On Mon 2026-08-31 @ 11:01:44 AM, Antonin Godard wrote:
>> >> Hi Quentin, Trevor,
>> >> 
>> >> After having thought about this a bit more, I think there are pros and cons to
>> >> both approaches, but I'm leaning towards Quentin's approach. The other approach
>> >> may feel "safe", and doesn't leave room to errors, but I'm afraid that 1. most
>> >> contributors will forget about it and 2. it might make the process more painful
>> >> for them.
>> >> 
>> >> So here what we should do:
>> >> 
>> >> - set `hightlight_language = "bitbake"` in conf.py.
>> >> 
>> >> - remove file-wide lexer enforcing (.. highlight:: directive at the top of the
>> >>   file) (in a separate patch)
>> >> 
>> >> - use the appropriate lexer for each code-block that is _not_ bitbake code.
>> >> 
>> >> - fix any parsing error from the bitbake lexer (I did have some when trying it).
>> >>   If this happens, use "none" and *add a identifiable comment* above it to
>> >>   explain that there's an issue with the lexer. This way we can track them and
>> >>   fix them when Pygments gets an update.
>> >
>> > The pygments releases occur on a rather slow timeline, I predict the next
>> > release will probably be in Dec if not Jan 2027, if history is any
>> > indicator.
>> >
>> > There's a problem (Quentin mentioned it in one of his replies): users
>> > don't use the tarball that the AB uses (in general, I assume) and
>> > versions are not pinned. So users are free to use whatever is on their
>> > system but hopefully have created a venv. But even if they're using a
>> > venv there's no guarantee that they're updating their tools regularly.
>> 
>> We can pin the version required to build the docs in
>> documentation/tools/host_packages_scripts/pip3_docs.sh. This will become the
>> minimum version of Pygments required to build the docs.
>> 
>> > So we're left with the following situation:
>> > - the AB has to wait until the tarball is updated
>> > - users might be using older versions of pygments
>> >
>> > And then on top of that you layer on the situation of trying to generate
>> > docs for older releases not to mention backports.
>> 
>> The change in conf.py will not impact older releases, so they should be safe to
>> build, even if Pygments is updated with the bitbake lexer: it simply won't be
>> used, because highlight_language is still "default" on these.
>
> If we set:
>
> 	highlight_language = "bitbake"
>
> in conf.py right now, anyone who is not running pygments 2.21.0 will get
> warnings which translate to build failures via the Makefile's -W
> setting.

We're on the master branch, having such changes is acceptable, and we can state
the version of pygments needed in our requirements.

>> Note: I think this change is big enough that I don't consider it candidate for
>> backport on stable branches.
>> 
>> > However, I can add a shim so that everything works out of the box
>> > today. The shim can be smart enough to examine the bitbake support
>> > independently (at runtime) and only load itself when it is needed
>> > (either an older version of pygments that has no support, or the 2.21.0
>> > version that needs additional support). If/when pygments is updated
>> > (after the next release and either because the user has updated their
>> > tools or the tarball has been updated) the shim will not load itself.
>> >
>> > Carrying a bitbake language shim in the docs repository itself:
>> > - the AB doesn't have to wait for an update, bitbake works today on all
>> >   valid snippets
>> > - backports can start working today too, since tooling doesn't have to
>> >   be updated, the shim knows how to highlight bitbake independent of
>> >   tools or versions
>> > - we don't have to wait for the next release (5-6 months) for all
>> >   the bitbake snippets to parse correctly
>> >
>> > The nice thing about the shim (the way I've designed it) is it is
>> > dynamic. It will look for any bitbake support in the currently used
>> > tools. After running a tiny bit of testing it will load itself only:
>> > - if there is no bitbake support
>> > - if the bitbake support is incomplete
>> > Otherwise it won't load and won't interfere.
>> >
>> > Even once the next release of pygments occurs and full bitbake support
>> > exists, it will still be a good idea to carry the shim so that older
>> > docs, users with older tools, and the AB doesn't have to wait for a new
>> > tarball to get full bitbake highlighting.
>> >
>> > Also, this way we don't have to skip the non-working snippets today
>> > with a comments, all snippets will highlight today, no need to go back
>> > and fix things up in 6 months if someone remembers, etc. Full bitbake
>> > highlighting for everyone under any circumstance starting today and
>> > available forever across all versions of docs and tools.
>> 
>> Considering what I've said above, do you really think that's necessary? To me it
>> feels like increased complexity considering we'll only add this feature on
>> master, and it should be straightforward from there to fix future stable
>> releases with backport patches when Pygments gets updated.
>
> What's the complexity? The shim is written, it's done, it works
> regardless of which pygments version anyone (AB, users) are using and it
> doesn't install if the conditions are met. No waiting, everyone gets all
> bitbake highlighted today.
>
> If anything not adding the shim *is* where the complexity lies. Once you
> accept the default conf.py patch the AB must be using the new pygments,
> and users will see build errors, and report them, until we point out
> they have to update their tools.

Please send it - perhaps as a separate patch - so I can review it.

Thanks,
Antonin