Re: [docs] [PATCH 00/10] docs: highlight BitBake snippets with the bitbake language

[Quentin Schulz <quentin.schulz@cherry.de>]

Hi Trevor,

On 8/26/26 9:56 PM, Trevor Woerner wrote:
> On Wed 2026-08-26 @ 04:45:33 PM, Quentin Schulz wrote:
>>
>>
>> On 8/26/26 3:25 PM, Trevor Woerner via lists.yoctoproject.org wrote:
>>> On Wed 2026-08-26 @ 03:09:23 PM, Antonin Godard wrote:
>>>> Hi,
>>>>
>>>> On Wed Aug 26, 2026 at 2:10 PM CEST, Quentin Schulz via lists.yoctoproject.org wrote:
>>>>> Hi Trevor,
>>>>>
>>>>> On 8/26/26 3:34 AM, Trevor Woerner via lists.yoctoproject.org wrote:[...]>>>
>>>>> [...]
>>>>>
>>>>>> Depends on
>>>>>> ----------
>>>>>>
>>>>>> "docs: state the language of nine literal blocks explicitly", sent
>>>>>
>>>>> Link to the ML please to make maintainers and reviewers job easier.
>>>>>
>>>>> [...]
>>>>>
>>>>> I think this is going the wrong direction. We should actually make
>>>>> explicit the language of every :: that is NOT to be understood as
>>>>> BitBake code and then make the default highlight language be BitBake.
>>>>
>>>> But then this might get forgotten? How about having the default highlighted as
>>>> "none", and make _everything_ use explicit code-blocks?
>>>>
>>>> Sure, this is more efforts and review time, but also this is how other markup
>>>> languages work - like markdown, where by default (when using ```...```) no
>>>> syntax highlighting is done.
>>>>
>>>> This is maybe a more conservative approach but at least it doesn't leave room
>>>> for code blocks mistakenly highlighted with the bitbake lexer.
>>>
>>> Yes, I agree too.  Personally I'm not fond of "hidden defaults".
>>>
>>
>> There's already one.
> 
> True. So the choice is between:
> - guessing python
> - guessing bitbake
> - or having the author state the expected language of every code-block
> 
> Personally my preference would be for every block to state it's expected
> language. I don't think that's either too onerous or redundant. We put
> ".bb" and ".bbclass" at the end of every bitbake and bitbake class file
> even though it's obvious what they are by virtue of where the live or
> what the contain. In general we put ".html" at the end of html files and
> ".txt" at the end of text files.
> 
> But assuming I can't convince everyone that the language should be
> explicit on each block, (and that each such item should have an explicit
> "code-block:: <lang>" tag) the discussion devolves into: which default
> is wrong least often :-) So i asked AI to measure it across all
> currently untagged blocks:
> 
> 
>                                  lexed   correct   wrong   unclass   plain   visibly
>                                          (known)  (known)                     wrong
>     ━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━  ━━━━━━━━  ━━━━━━━  ━━━━━━━━  ━━━━━━  ━━━━━━━
>      yocto-docs
>        bitbake                    1502       841        9       652       0       96
>        python (today)              782         5      635       142     720      777
>        none                          0         0        0         0    1502        0
>     ─────────────────────────  ───────  ────────  ───────  ────────  ──────  ───────
>      BitBake user manual
>        bitbake                     299       193        6       100       0       16
>        python (today)              182         6      148        28     117      176
>        none                          0         0        0         0     299        0
>     ─────────────────────────  ───────  ────────  ───────  ────────  ──────  ───────
>      both
>        bitbake                    1801      1034       15       752       0      112
>        python (today)              964        11      783       170     837      953
>        none                          0         0        0         0    1801        0
> 
> 
> Setting the default to bitbake would be a real improvement on what we
> have today. But it doesn't get us to correct. Across both repos it is
> still wrong on 15 blocks we can name, it has no opinion at all about
> another 752, and 112 end up visibly wrong. Those 752 won't be right
> regardless of which default we choose.
> 
> Assuming your proposal is to set bitbake as the default, then tag
> everything else. That _sounds_ good:
> 
>                              untagged   bitbake   still need   % of
>                                blocks               a tag      untagged
>      ━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━  ━━━━━━━━  ━━━━━━━━━━  ━━━━━━━━━━
>       yocto-docs                  1502       841         661        44%
>      ─────────────────────────  ────────  ────────  ──────────  ──────────
>       BitBake user manual          299       193         106        35%
>      ─────────────────────────  ────────  ────────  ──────────  ──────────
>       both                        1801      1034         767        43%
> 
> 
> Under "default bitbake, mark the exceptions" we would still have to
> explicitly tag 661 blocks in yocto-docs and 106 in the bitbake manual:
> 767 by hand. Tagging everything is 1801. So we are marking hundreds of
> blocks either way. 43% is not an "exception".
> 
> Which leaves how each scheme fails when somebody forgets. With bitbake as
> the default a missed block renders as bitbake - confidently and silently
> wrong. With none as the default it renders plain: bare, but it doesn't
> claim anything untrue. Given we're tagging hundreds of blocks regardless,
> I'd rather the mistakes look unstyled than look wrong.
> 

Just to be clear, we do not use none today. We use "default", which uses 
Python except if there are warnings, in which case it falls back to none.

> That matters more here than it would in most projects, because bitbake is
> a genuinely bad language to guess mechanically. Shell sessions,
> TEMPLATECONF= environment assignments and buildhistory output all share
> bitbake's NAME = value shape, so they pick up variable-assignment
> colouring without being bitbake at all. In a lot of places our snippets
> look like console, or bash, or python - or maybe bitbake.
> 
>>> By default a non-specified block will cause pygments to guess. So if I'm
>>
>> Sphinx explicitly states that Python will be tried and if there are
>> highlights warnings, none will be used. c.f. https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#showing-code-examples
>>
>>> looking at a lot of reST and seeing no tags, I'm going to assume the
>>> authors have left pygments to guess. I would have to go looking in other
>>> places to see if there's a default in place for non-specified code-block
>>> tags, which I'm probably not going to think of doing, which will
>>> sometimes probably cause more confusion before it becomes clear.
>>>
>>
>> This is the Yocto/BitBake docs, we're documenting that project, which uses
>> BitBake syntax. Would you want to write
>>
>> .. code-block:: rust
>>
>> for every code-block in the Rust documentation?
> 
> Yes.
> 

I find this very unnecessary but I guess we can agree to disagree :)

> First off, not all blocks are going to be rust, just as not all of ours
> are bitbake:
> 
>                            Total blocks    BitBake    of all    of untagged
>    ━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━  ━━━━━━━━━  ━━━━━━━━  ━━━━━━━━━━━━━
>     yocto-docs                     1929        841       44%            56%
>    ─────────────────────  ──────────────  ─────────  ────────  ─────────────
>     BitBake user manual             334        193       58%            65%
> 
> 
> You might also be suspicious of an LLM's ability to judge what language
> a block is. So am I. So the classification isn't what the numbers rest
> on: every block that got tagged was then fed to BitBake's own parser,
> using the regexes out of ConfHandler and BBHandler, and anything the
> parser rejected was re-examined. That's how a Makefile, three U-Boot FIT
> source blocks, five Python unittest classes and five kernel .scc files
> got caught and pulled back out of an earlier version of this series.
> The counts above come from parsing the built HTML.
> 

I actually forced a code block to be Python and attempted to make it 
syntactically wrong, the lexer still happily chugged along. So I'm not 
sure any tool will get the numbers right.

>>> Maybe it's just me, but I prefer to see every example block using an
>>> explicit code-block tag and specifying a language. That way the author's
>>> intentions are unambiguous, and we have an expected output.
>>>
>>
>> You won't be able to backport patches with
>>
>> .. code-block:: bitbake
>>
>> which is what I would like to avoid. But I'm not the one sending patches to
>> the stable branches nor the maintainer(s) handling stable, so maybe I
>> shouldn't care.
> 
> For everything going back to scarthgap, the auto-builder uses the same
> tarball for building the docs, regardless of branch. Therefore backports
> amongst everything from scarthgap to master would be fine, once that
> tarball is updated.
> 

This is looked at through the autobuilder scope, the docs are still 
supposed to be somewhat buildable without buildtools or autobuilders. 
but fine.

> For backports to branches earlier than scarthgap the AB doesn't use the
> -W flag, so those would simply emit a warning and render the blocks
> plain. I checked this against the actual pairing in the older tarball,
> sphinx 5.1.1 with pygments 2.13.0:
> 
> 	WARNING: Pygments lexer name 'bitbake' is not known
> 	build succeeded, 1 warning.
> 
> with the block rendering plain. So that's not breakage, it's plain
> presentation.
> 

I'm not sure saying warnings are fine is something the project wants to 
get behind. But maybe it is fine, I don't have much power there :)

> And it's arguably better than what those branches show today. Of the 841
> blocks this series tags as bitbake, 632 currently render *highlighted
> as python* and only 209 render plain. So on an older branch a backport
> would be trading incorrect highlighting for none, rather than losing
> anything.
> 
> 
>>> That's what I've enforced for our reST at work, and I've also
>>> differentiated between "bash" (for actual shell programs) versus
>>> "console" (for interactive work at a shell prompt, a shell most likely
>>> to be bash).
>>>
>>
>> We differentiate console and shell in these docs here too. If you're seeing
>> one that is using the wrong lexer, please fix it.
>>
>> Cheers,
>> Quentin