carrying lexers in documentation/sphinx: the ones after the shim

[Trevor Woerner <twoerner@gmail.com>]

Hi Antonin,

This work started as a desire to tag and highlight BitBake code-blocks
in our own documentation. Along the way it evolved into a task to
correctly label (and highlight) all code-blocks throughout the Yocto
Project documentation. In order to perform this work, two prerequisites
had to be met: every code-block had to be valid for whichever language
was appropriate, and the necessary lexers had to be added. Thank you for 
taking the first; now let's move on to the next.

A while back, you asked me to send the BitBake shim as its own patch. I
will. I am proposing a set of additional lexer patches, and wanted to
discuss them before sending.

One definition first, since the following table uses it. A "placeholder"
is a value the reader is expected to replace with one of their own,
written in angle brackets. E.g.:

   SRCREV_<nameX> = "<revision SHA for nameX>"

That line is not valid BitBake - the charset rejects "<" in a name - so
the block cannot honestly be tagged "bitbake". Rather than teach the 
BitBake lexer to accept "<" in names everywhere, including in the blocks
that are real metadata, "-placeholder" names a dialect: never a language
on its own, always a parent plus placeholders. It extends the parent
rather than restating it, and a code-block opts in only if it needs it. 

  lexer                     lines  derived from
  ------------------------  -----  ---------------------------
  bitbake (the shim)          442  BitBake's compiled regexes
  bitbake-placeholder         147  the above, one rule relaxed
  console (corrected)          53  a rule the manuals already keep
  console-placeholder,
    text-placeholder          140  a filter over the parent's tokens
  scc                         117  yocto-kernel-tools
  wks                          67  wic's ksparser.py
  pydevshell                   48  devshell.bbclass's prompt

Upstream is not the home for these. Pygments' contributing guide expects
a language "to have a reasonably sized community around it" and suggests
a plugin otherwise; ".scc" and ".wks" are very small, Yocto-specific
formats nobody outside this project writes, so that is not a bar they
will ever clear.

The corrected console lexer changes a rendering we already have, so
it is worth flagging. Pygments lets the preamble before a prompt be
empty, so a line opening with "#" is read as a root prompt and the rest
lexed as a command. Checking the console blocks in the manuals, I find
every such line is a comment or program output and not one is a prompt.
Requiring the preamble is the whole diff; "$" and "%" are untouched.

Remaining items on my plate:

  - the shim, on its own, as you asked
  - the lexers (discussed above), one patch each
  - the labelling and the conf.py flip last

If it makes the discussion easier, I could build and send a sample of
the generated HTML docs after these changes, so you can see how the 
result will look instead of having to judge based on a description
alone.

Best regards,
        Trevor