* [RFC PATCH 1/2] add a "bitbake" pygments lexer
2026-04-21 17:24 [RFC PATCH 0/2] support bitbake blocks Trevor Woerner
@ 2026-04-21 17:24 ` Trevor Woerner
2026-04-22 6:55 ` [docs] " Antonin Godard
2026-04-21 17:24 ` [RFC PATCH 2/2] BSP dev guide: update to use bitbake lexer Trevor Woerner
2026-04-21 19:36 ` [RFC PATCH 0/2] support bitbake blocks Trevor Woerner
2 siblings, 1 reply; 8+ messages in thread
From: Trevor Woerner @ 2026-04-21 17:24 UTC (permalink / raw)
To: docs; +Cc: Trevor Woerner
From: Trevor Woerner <trevor.woerner@amd.com>
Highlighting renders correctly across the BitBake constructs from
the upstream vim syntax:
comments
all assignment operators (=, ?=, +=, =+, :=, ??=, .=, =.)
${VAR} and ${@...} interpolation
OE override chains (FILES:${PN}-doc, :append:class-target)
varflags (do_install[depends])
inherit/include/require
addtask ... after ... before ...
EXPORT_FUNCTIONS
shell function bodies (delegated to BashLexer)
python/fakeroot python task bodies and top-level def blocks (delegated
to PythonLexer).
Changes
documentation/sphinx/bitbake.py is a new local Sphinx extension that
defines BitbakeLexer (a Pygments RegexLexer) and registers it with
Sphinx via sphinx.highlighting.lexers under the aliases bitbake and bb.
Token rules mirror contrib/vim/syntax/bitbake.vim from the bitbake repo,
and inner shell/python regions are dispatched to the existing BashLexer
/ PythonLexer using Pygments' using() helper.
documentation/conf.py appends 'bitbake' to extensions so the lexer is
loaded on every build.
Usage
In any .rst file you can now write:
.. code-block:: bitbake
SUMMARY = "Hello"
inherit autotools
do_install:append() {
install -d ${D}${bindir}
}
bb works as an alias if shorter is preferred, and the same applies to
.. highlight:: bitbake.
AI-Generated: codex/claude-opus 4.7 (xhigh)
Signed-off-by: Trevor Woerner <trevor.woerner@amd.com>
---
documentation/conf.py | 3 +-
documentation/sphinx/bitbake.py | 195 ++++++++++++++++++++++++++++++++
2 files changed, 197 insertions(+), 1 deletion(-)
create mode 100644 documentation/sphinx/bitbake.py
diff --git a/documentation/conf.py b/documentation/conf.py
index e5e3a8a89abd..aa2ec8d45ed3 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -68,7 +68,8 @@ extensions = [
'sphinx.ext.intersphinx',
'sphinx_copybutton',
'sphinxcontrib.rsvgconverter',
- 'yocto-vars'
+ 'yocto-vars',
+ 'bitbake',
]
autosectionlabel_prefix_document = True
diff --git a/documentation/sphinx/bitbake.py b/documentation/sphinx/bitbake.py
new file mode 100644
index 000000000000..c521340c391f
--- /dev/null
+++ b/documentation/sphinx/bitbake.py
@@ -0,0 +1,195 @@
+#!/usr/bin/env python
+#
+# SPDX-License-Identifier: CC-BY-SA-2.0-UK
+#
+# Pygments lexer for BitBake metadata files (``.bb``, ``.bbclass``,
+# ``.bbappend``, ``.inc``, ``.conf``) and inline ``code-block:: bitbake``
+# snippets used throughout the Yocto Project documentation.
+#
+# Pygments does not ship a BitBake lexer, so this Sphinx extension provides
+# one. The token rules below mirror the upstream BitBake vim syntax shipped
+# in ``contrib/vim/syntax/bitbake.vim`` of the bitbake repository:
+# https://git.openembedded.org/bitbake/tree/contrib/vim/syntax/bitbake.vim
+#
+# Highlights:
+# - comments, line continuations, varflags
+# - ``${VAR}`` and ``${@python_expr}`` interpolations
+# - assignment operators (``=``, ``:=``, ``+=``, ``=+``, ``.=``, ``=.``,
+# ``?=``, ``??=``) with optional ``export`` prefix and OE override
+# suffixes (``VAR:append``, ``FILES:${PN}-doc``...)
+# - ``inherit``, ``include``, ``require`` directives
+# - ``addtask`` / ``deltask`` / ``addhandler`` / ``EXPORT_FUNCTIONS``
+# statements (with the ``after`` / ``before`` keywords)
+# - shell task bodies (``do_install() { ... }``) delegated to ``BashLexer``
+# - ``python``, ``fakeroot python``, anonymous ``python() { ... }`` task
+# bodies and top-level ``def`` blocks delegated to ``PythonLexer``
+
+import re
+
+from pygments.lexer import RegexLexer, bygroups, include, using, words
+from pygments.lexers.python import PythonLexer
+from pygments.lexers.shell import BashLexer
+from pygments.token import (
+ Comment,
+ Keyword,
+ Name,
+ Operator,
+ Punctuation,
+ String,
+ Text,
+ Whitespace,
+)
+
+__version__ = '1.0'
+
+# A bare BitBake identifier (variable, function or flag name). Allows the
+# characters used by OE-Core variable names (digits, ``-``, ``.``, ``+``).
+_IDENT = r'[A-Za-z_][A-Za-z0-9_\-.+]*'
+
+# Optional OE override chain such as ``:append``, ``:remove``, ``:class-target``
+# or ``:${PN}-doc``. Anchored so it only consumes ``:foo`` runs and never
+# eats the leading ``:`` of the ``:=`` assignment operator.
+_OVERRIDE = r'(?::[A-Za-z0-9_\-.+${}]+)*'
+
+# All BitBake variable assignment operators, ordered so that the longer
+# operators win the regex alternation.
+_ASSIGN = r'(?:\?\?=|\?=|:=|\+=|=\+|\.=|=\.|=)'
+
+
+class BitbakeLexer(RegexLexer):
+ """Lexer for BitBake recipes, classes, includes and configuration."""
+
+ name = 'BitBake'
+ aliases = ['bitbake', 'bb']
+ filenames = ['*.bb', '*.bbclass', '*.bbappend', '*.inc', '*.conf']
+ mimetypes = ['text/x-bitbake']
+
+ flags = re.MULTILINE
+
+ tokens = {
+ 'root': [
+ (r'[ \t]+', Whitespace),
+ (r'\n', Whitespace),
+ (r'#.*$', Comment.Single),
+
+ # ``python [name]() { ... }`` blocks (also ``fakeroot python``).
+ # Must be tried before the generic shell function rule so the
+ # ``python`` keyword is not mistaken for a shell function name.
+ (r'(^(?:fakeroot[ \t]+)?)(python)((?:[ \t]+' + _IDENT + r')?)'
+ r'([ \t]*\([ \t]*\)[ \t]*)(\{[ \t]*\n)'
+ r'((?:.*\n)*?)'
+ r'(^\}[ \t]*$)',
+ bygroups(Keyword.Type, Keyword, Name.Function, Text,
+ Punctuation, using(PythonLexer), Punctuation)),
+
+ # Shell task bodies: ``[fakeroot ]name[:override]() { ... }``.
+ (r'(^(?:fakeroot[ \t]+)?)(' + _IDENT + r')(' + _OVERRIDE + r')'
+ r'([ \t]*\([ \t]*\)[ \t]*)(\{[ \t]*\n)'
+ r'((?:.*\n)*?)'
+ r'(^\}[ \t]*$)',
+ bygroups(Keyword.Type, Name.Function, Name.Decorator, Text,
+ Punctuation, using(BashLexer), Punctuation)),
+
+ # Top-level python ``def`` blocks; the body is any run of
+ # indented or blank lines following the signature.
+ (r'^def[ \t]+' + _IDENT + r'[ \t]*\([^)]*\)[ \t]*:[ \t]*\n'
+ r'(?:[ \t]+.*\n|\n)+',
+ using(PythonLexer)),
+
+ # ``inherit`` / ``include`` / ``require`` directives.
+ (r'^(inherit|include|require)\b',
+ Keyword.Namespace, 'include-line'),
+
+ # ``addtask`` / ``deltask`` / ``addhandler`` / ``EXPORT_FUNCTIONS``.
+ (r'^(addtask|deltask|addhandler|EXPORT_FUNCTIONS)\b',
+ Keyword, 'statement'),
+
+ # ``VAR[flag] = "value"`` (varflag assignment).
+ (r'^(' + _IDENT + r')(\[)(' + _IDENT + r')(\])([ \t]*)('
+ + _ASSIGN + r')',
+ bygroups(Name.Variable, Punctuation, Name.Attribute,
+ Punctuation, Whitespace, Operator),
+ 'value'),
+
+ # ``[export ]VAR[:override...] OP "value"`` assignments.
+ (r'^(export[ \t]+)?(' + _IDENT + r')(' + _OVERRIDE + r')'
+ r'([ \t]*)(' + _ASSIGN + r')',
+ bygroups(Keyword.Type, Name.Variable, Name.Decorator,
+ Whitespace, Operator),
+ 'value'),
+
+ # Anything else: fall through one character at a time.
+ (r'.', Text),
+ ],
+
+ 'include-line': [
+ (r'[ \t]+', Whitespace),
+ (r'\\\n', Text),
+ (r'\n', Whitespace, '#pop'),
+ include('interp'),
+ (r'[^\s$]+', String),
+ ],
+
+ 'statement': [
+ (r'[ \t]+', Whitespace),
+ (r'\\\n', Text),
+ (r'\n', Whitespace, '#pop'),
+ (words(('after', 'before'), suffix=r'\b'), Keyword),
+ include('interp'),
+ (r'[^\s$\\]+', Name),
+ ],
+
+ 'value': [
+ (r'[ \t]+', Whitespace),
+ (r'\\\n', String.Escape),
+ (r'\n', Whitespace, '#pop'),
+ (r'"', String.Double, 'string-double'),
+ (r"'", String.Single, 'string-single'),
+ include('interp'),
+ (r'[^\s"\'$\\]+', String),
+ ],
+
+ 'string-double': [
+ (r'\\\n', String.Escape),
+ (r'\\.', String.Escape),
+ (r'"', String.Double, '#pop'),
+ include('interp'),
+ (r'[^"\\$]+', String.Double),
+ ],
+
+ 'string-single': [
+ (r'\\\n', String.Escape),
+ (r'\\.', String.Escape),
+ (r"'", String.Single, '#pop'),
+ include('interp'),
+ (r"[^'\\$]+", String.Single),
+ ],
+
+ 'interp': [
+ # ``${@ python expression }`` evaluated by BitBake at parse time.
+ (r'\$\{@', String.Interpol, 'py-interp'),
+ # ``${VAR}`` variable expansion.
+ (r'(\$\{)([A-Za-z0-9_\-:.+/]+)(\})',
+ bygroups(String.Interpol, Name.Variable, String.Interpol)),
+ ],
+
+ 'py-interp': [
+ (r'\}', String.Interpol, '#pop'),
+ (r'[^}]+', using(PythonLexer)),
+ ],
+ }
+
+
+def setup(app):
+ """Register the BitBake lexer with Sphinx/Pygments."""
+ from sphinx.highlighting import lexers
+
+ bb_lexer = BitbakeLexer()
+ lexers['bitbake'] = bb_lexer
+ lexers['bb'] = bb_lexer
+
+ return {
+ 'version': __version__,
+ 'parallel_read_safe': True,
+ 'parallel_write_safe': True,
+ }
--
2.50.0.173.g8b6f19ccfc3a
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC PATCH 2/2] BSP dev guide: update to use bitbake lexer
2026-04-21 17:24 [RFC PATCH 0/2] support bitbake blocks Trevor Woerner
2026-04-21 17:24 ` [RFC PATCH 1/2] add a "bitbake" pygments lexer Trevor Woerner
@ 2026-04-21 17:24 ` Trevor Woerner
2026-04-21 19:36 ` [RFC PATCH 0/2] support bitbake blocks Trevor Woerner
2 siblings, 0 replies; 8+ messages in thread
From: Trevor Woerner @ 2026-04-21 17:24 UTC (permalink / raw)
To: docs; +Cc: Trevor Woerner
From: Trevor Woerner <trevor.woerner@amd.com>
Identify all bitbake blocks, re-categorize them as "code-blocks" and set
their language as "bitbake".
10 highlighted BitBake blocks emitted in the rendered HTML, matching
the 10 code-block:: bitbake directives now in bsp.rst. Build completed
with the same 189 pre-existing warnings (all intersphinx / cross- ref
related, none from these changes).
Edits
In documentation/bsp-guide/bsp.rst, I converted every snippet that
is actually BitBake metadata from a plain literal block (::) to ..
code-block:: bitbake. Shell transcripts, directory listings, and prose-
following :: blocks were left alone. The 10 converted blocks are:
1. BBLAYERS ?= ... example near the BSP-layer overview.
2. The expanded BBLAYERS example covering container layers like
meta-openembedded/meta-oe.
3. The boilerplate conf/layer.conf (BBPATH, BBFILES,
BBFILE_COLLECTIONS, LAYERDEPENDS_bsp).
4. The Raspberry Pi conf/layer.conf excerpt with LICENSE_PATH.
5. The include conf/machine/include/rpi-base.inc directive in
raspberrypi3.conf.
6. The PREFERRED_PROVIDER_virtual/kernel /
PREFERRED_VERSION_linux-yocto snippet in the kernel-recipe section.
7. The FILESEXTRAPATHS:prepend := "${THISDIR}/files:" example in the
customizing-a-recipe walkthrough.
8. The Beaglebone conf/layer.conf shown in "BSP Layer Configuration
Example".
9. The Beaglebone PREFERRED_PROVIDER_virtual/kernel /
PREFERRED_VERSION_linux-yocto snippet in "BSP Kernel Recipe Example".
10. The linux-yocto_6.1.bbappend body (KBRANCH:*, KMACHINE:*,
SRCREV_machine:*, COMPATIBLE_MACHINE:*, LINUX_VERSION:*).
The directory-listing block at line 290 stays as code-block:: none
(it is a filesystem tree, not BitBake), and the formfactor machconfig
snippet (HAVE_TOUCHSCREEN=0 etc.) is intentionally left as a plain
literal block since it is shell-style key=value, not BitBake syntax.
Verification
make (python3 -m sphinx -b html ... . _build/html) completes with
exit 0 and build succeeded, 189 warnings, identical to the pre-change
baseline; none of those warnings reference bsp.rst line numbers I
touched, the new bitbake extension, or Pygments lexing. Spot-checking
the rendered HTML shows 10 highlight-bitbake Pygments-highlighted
blocks in documentation/_build/html/bsp-guide/bsp.html, matching the 10
directives.
AI-Generated: codex/claude-opus 4.7 (xhigh)
Signed-off-by: Trevor Woerner <trevor.woerner@amd.com>
---
documentation/bsp-guide/bsp.rst | 40 ++++++++++++++++++++++++---------
1 file changed, 30 insertions(+), 10 deletions(-)
diff --git a/documentation/bsp-guide/bsp.rst b/documentation/bsp-guide/bsp.rst
index a3b57d29f9e6..67dd9a67d651 100644
--- a/documentation/bsp-guide/bsp.rst
+++ b/documentation/bsp-guide/bsp.rst
@@ -84,7 +84,9 @@ established after you run the OpenEmbedded build environment setup
script (i.e. :ref:`structure-core-script`).
Adding the root directory allows the :term:`OpenEmbedded Build System`
to recognize the BSP
-layer and from it build an image. Here is an example::
+layer and from it build an image. Here is an example:
+
+.. code-block:: bitbake
BBLAYERS ?= " \
/usr/local/src/yocto/meta \
@@ -112,7 +114,9 @@ are known as ":term:`container layers <Container Layer>`". An example of
this type of layer is OpenEmbedded's :oe_git:`meta-openembedded </meta-openembedded>`
layer. The ``meta-openembedded`` layer contains many ``meta-*`` layers.
In cases like this, you need to include the names of the actual layers
-you want to work with, such as::
+you want to work with, such as:
+
+.. code-block:: bitbake
BBLAYERS ?= " \
/usr/local/src/yocto/meta \
@@ -531,7 +535,9 @@ identifies the contents of the layer, and contains information about how
the build system should use it. Generally, a standard boilerplate file
such as the following works. In the following example, you would replace
"bsp" with the actual name of the BSP (i.e. "bsp_root_name" from the example
-template). ::
+template).
+
+.. code-block:: bitbake
# We have a conf and classes directory, add to BBPATH
BBPATH .= ":${LAYERDIR}"
@@ -546,7 +552,9 @@ template). ::
LAYERDEPENDS_bsp = "intel"
To illustrate the string substitutions, here are the corresponding
-statements from the Raspberry Pi ``conf/layer.conf`` file::
+statements from the Raspberry Pi ``conf/layer.conf`` file:
+
+.. code-block:: bitbake
# We have a conf and classes directory, append to BBPATH
BBPATH .= ":${LAYERDIR}"
@@ -603,7 +611,9 @@ For example, many ``tune-*`` files (e.g. ``tune-arm1136jf-s.inc``,
To use an include file, you simply include them in the machine
configuration file. For example, the Raspberry Pi BSP
-``raspberrypi3.conf`` contains the following statement::
+``raspberrypi3.conf`` contains the following statement:
+
+.. code-block:: bitbake
include conf/machine/include/rpi-base.inc
@@ -674,7 +684,9 @@ Suppose you are using the ``linux-yocto_6.12.bb`` recipe to build the
kernel. In other words, you have selected the kernel in your
``"bsp_root_name".conf`` file by adding
:term:`PREFERRED_PROVIDER` and :term:`PREFERRED_VERSION`
-statements as follows::
+statements as follows:
+
+.. code-block:: bitbake
PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
PREFERRED_VERSION_linux-yocto ?= "6.12%"
@@ -1027,7 +1039,9 @@ BSP-specific configuration file named ``interfaces`` to the
also supports several other machines:
#. Edit the ``init-ifupdown_1.0.bbappend`` file so that it contains the
- following::
+ following:
+
+ .. code-block:: bitbake
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
@@ -1195,7 +1209,9 @@ BSP Layer Configuration Example
-------------------------------
The layer's ``conf`` directory contains the ``layer.conf`` configuration
-file. In this example, the ``conf/layer.conf`` file is the following::
+file. In this example, the ``conf/layer.conf`` file is the following:
+
+.. code-block:: bitbake
# We have a conf and classes directory, add to BBPATH
BBPATH .= ":${LAYERDIR}"
@@ -1375,7 +1391,9 @@ BSP Kernel Recipe Example
-------------------------
The kernel recipe used to build the kernel image for the BeagleBone
-device was established in the machine configuration::
+device was established in the machine configuration:
+
+.. code-block:: bitbake
PREFERRED_PROVIDER_virtual/kernel ?= "linux-yocto"
PREFERRED_VERSION_linux-yocto ?= "6.1%"
@@ -1386,7 +1404,9 @@ metadata used to build the kernel. In this case, a kernel append file
kernel recipe (i.e. ``linux-yocto_6.1.bb``), which is located in
:oe_git:`/openembedded-core/tree/meta/recipes-kernel/linux`.
-The contents of the append file are::
+The contents of the append file are:
+
+.. code-block:: bitbake
KBRANCH:genericx86 = "v6.1/standard/base"
KBRANCH:genericx86-64 = "v6.1/standard/base"
--
2.50.0.173.g8b6f19ccfc3a
^ permalink raw reply related [flat|nested] 8+ messages in thread