carrying lexers in documentation/sphinx: the ones after the shim

carrying lexers in documentation/sphinx: the ones after the shim

[Trevor Woerner]

Hi Antonin,

This work started as a desire to tag and highlight BitBake code-blocks
in our own documentation. Along the way it evolved into a task to
correctly label (and highlight) all code-blocks throughout the Yocto
Project documentation. In order to perform this work, two prerequisites
had to be met: every code-block had to be valid for whichever language
was appropriate, and the necessary lexers had to be added. Thank you for 
taking the first; now let's move on to the next.

A while back, you asked me to send the BitBake shim as its own patch. I
will. I am proposing a set of additional lexer patches, and wanted to
discuss them before sending.

One definition first, since the following table uses it. A "placeholder"
is a value the reader is expected to replace with one of their own,
written in angle brackets. E.g.:

   SRCREV_<nameX> = "<revision SHA for nameX>"

That line is not valid BitBake - the charset rejects "<" in a name - so
the block cannot honestly be tagged "bitbake". Rather than teach the 
BitBake lexer to accept "<" in names everywhere, including in the blocks
that are real metadata, "-placeholder" names a dialect: never a language
on its own, always a parent plus placeholders. It extends the parent
rather than restating it, and a code-block opts in only if it needs it. 

  lexer                     lines  derived from
  ------------------------  -----  ---------------------------
  bitbake (the shim)          442  BitBake's compiled regexes
  bitbake-placeholder         147  the above, one rule relaxed
  console (corrected)          53  a rule the manuals already keep
  console-placeholder,
    text-placeholder          140  a filter over the parent's tokens
  scc                         117  yocto-kernel-tools
  wks                          67  wic's ksparser.py
  pydevshell                   48  devshell.bbclass's prompt

Upstream is not the home for these. Pygments' contributing guide expects
a language "to have a reasonably sized community around it" and suggests
a plugin otherwise; ".scc" and ".wks" are very small, Yocto-specific
formats nobody outside this project writes, so that is not a bar they
will ever clear.

The corrected console lexer changes a rendering we already have, so
it is worth flagging. Pygments lets the preamble before a prompt be
empty, so a line opening with "#" is read as a root prompt and the rest
lexed as a command. Checking the console blocks in the manuals, I find
every such line is a comment or program output and not one is a prompt.
Requiring the preamble is the whole diff; "$" and "%" are untouched.

Remaining items on my plate:

  - the shim, on its own, as you asked
  - the lexers (discussed above), one patch each
  - the labelling and the conf.py flip last

If it makes the discussion easier, I could build and send a sample of
the generated HTML docs after these changes, so you can see how the 
result will look instead of having to judge based on a description
alone.

Best regards,
        Trevor

Re: [docs] carrying lexers in documentation/sphinx: the ones after the shim

[Antonin Godard]

Hi,

On Wed Oct 21, 2026 at 5:17 PM CEST, Trevor Woerner via lists.yoctoproject.org wrote:
> Hi Antonin,
>
> This work started as a desire to tag and highlight BitBake code-blocks
> in our own documentation. Along the way it evolved into a task to
> correctly label (and highlight) all code-blocks throughout the Yocto
> Project documentation. In order to perform this work, two prerequisites
> had to be met: every code-block had to be valid for whichever language
> was appropriate, and the necessary lexers had to be added. Thank you for 
> taking the first; now let's move on to the next.
>
> A while back, you asked me to send the BitBake shim as its own patch. I
> will. I am proposing a set of additional lexer patches, and wanted to
> discuss them before sending.
>
> One definition first, since the following table uses it. A "placeholder"
> is a value the reader is expected to replace with one of their own,
> written in angle brackets. E.g.:
>
>    SRCREV_<nameX> = "<revision SHA for nameX>"
>
> That line is not valid BitBake - the charset rejects "<" in a name - so
> the block cannot honestly be tagged "bitbake". Rather than teach the 
> BitBake lexer to accept "<" in names everywhere, including in the blocks
> that are real metadata, "-placeholder" names a dialect: never a language
> on its own, always a parent plus placeholders. It extends the parent
> rather than restating it, and a code-block opts in only if it needs it. 
>
>   lexer                     lines  derived from
>   ------------------------  -----  ---------------------------
>   bitbake (the shim)          442  BitBake's compiled regexes

When you say "the shim" I'm not entirely clear whether it will use the upstream
Pygments lexer or will completely override it? Can't we use the upstream lexer
for these one? Maybe I got the wrong idea, maybe seeing the code will clear
things up.

>   bitbake-placeholder         147  the above, one rule relaxed
>   console (corrected)          53  a rule the manuals already keep
>   console-placeholder,
>     text-placeholder          140  a filter over the parent's tokens
>   scc                         117  yocto-kernel-tools
>   wks                          67  wic's ksparser.py
>   pydevshell                   48  devshell.bbclass's prompt
>
> Upstream is not the home for these. Pygments' contributing guide expects
> a language "to have a reasonably sized community around it" and suggests
> a plugin otherwise; ".scc" and ".wks" are very small, Yocto-specific
> formats nobody outside this project writes, so that is not a bar they
> will ever clear.

For all but the regular bitbake lexer, I also share the opinion these should not
be upstream.

> The corrected console lexer changes a rendering we already have, so
> it is worth flagging. Pygments lets the preamble before a prompt be
> empty, so a line opening with "#" is read as a root prompt and the rest
> lexed as a command. Checking the console blocks in the manuals, I find
> every such line is a comment or program output and not one is a prompt.
> Requiring the preamble is the whole diff; "$" and "%" are untouched.

Not sure I fully understand what you mean here, but seeing the code might clear
that up too.

> Remaining items on my plate:
>
>   - the shim, on its own, as you asked
>   - the lexers (discussed above), one patch each
>   - the labelling and the conf.py flip last

Sounds like a plan!

> If it makes the discussion easier, I could build and send a sample of
> the generated HTML docs after these changes, so you can see how the 
> result will look instead of having to judge based on a description
> alone.

It'll be fairly easy for me to build and see it for myself I guess, so I don't
think you have to bother.

Thanks!
Antonin