[PATCH v4 0/3] migrate Debian/Ubuntu to pip packages for docs + fix epub and latexpdf targets not finding glob images

[PATCH v4 0/3] migrate Debian/Ubuntu to pip packages for docs + fix epub and latexpdf targets not finding glob images

[Quentin Schulz]

This fixes images being improperly handled by our Make targets by
switching to a sphinx extension that generates a PNG or PDF of an SVG
upon request by Sphinx itself.

To do that, we need sphinxcontrib-svg2pdfconverter>=2.0.0 which is not
available for all supported distros, including Debian/Ubuntu which only
used packages from their official package feeds until now. This migrates
both distros to use pip packages for building the docs such that the
same instructions are used for all supported distros.
This however required to use a virtualenv to install packages, since
Debian doesn't allow using pip outside of a virtualenv since Bookworm
(12). So this migrates the pip package installation to use a virtualenv
as well. Since sourcing scripts in RUN commands to make variables
accessible in later container layers is not possible in Containerfile,
we do the same logic as in the venv bin/activate script manually in the
Containerfile.

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
Changes in v4:
- migrate all distros to use virtualenv for installing packages via pip,
- migrate Debian and Ubuntu to use pip installation steps for building
  the docs,
- tested all distros build correctly by running the `html epub` make
  targets (`all` doesn't work for almalinux, rockylinux and centos
  stream at the moment),
- Link to v3: https://patch.msgid.link/20260211-fix-make-multi-target-v3-1-1168a6f1525f@cherry.de

Changes in v3:
- rebased on top of master, rebuilt with `make all` no build error to be
  seen,
- Link to v2: https://patch.msgid.link/20251202-fix-make-multi-target-v2-1-7eea1d64cf6b@cherry.de

Changes in v2:
- switched to upstream sphinxcontrib-svg2pdfconverter since the
  necessary MR got merged and a new release made,
- removed IMAGEDIRS Make variable as it is now unused,
- added sphinxcontrib-svg2pdfconverter to Pipfile (not tested) and pip
  installation script,
- Link to v1: https://patch.msgid.link/20251030-fix-make-multi-target-v1-0-213616ed1f0a@cherry.de

---
Quentin Schulz (3):
      use a venv for installing packages with pip
      consistently use pip packages for all distros
      convert SVGs to PDF and PNG using sphinxcontrib.rsvgconverter plugin

 documentation/Makefile                             | 27 +++-------------------
 documentation/Pipfile                              |  2 ++
 documentation/conf.py                              |  1 +
 documentation/ref-manual/system-requirements.rst   |  3 +++
 documentation/tools/build-docs-container           |  4 +++-
 .../tools/containerfiles/Containerfile.apt         | 10 ++++++++
 .../tools/containerfiles/Containerfile.dnf         |  6 +++++
 .../tools/containerfiles/Containerfile.zypper      |  7 ++++++
 .../tools/host_packages_scripts/pip3_docs.sh       |  4 +++-
 .../tools/host_packages_scripts/ubuntu_docs.sh     |  2 +-
 10 files changed, 39 insertions(+), 27 deletions(-)
---
base-commit: 095981c08b9d63905472df5d1d60c07af96f0250
change-id: 20251029-fix-make-multi-target-d2de9d4ff7ec

Best regards,
--  
Quentin Schulz <quentin.schulz@cherry.de>

[PATCH v4 1/3] use a venv for installing packages with pip

[Quentin Schulz]

From: Quentin Schulz <quentin.schulz@cherry.de>

In a few commits, we'll migrate Debian/Ubuntu to use pip packages.
However, Debian rejects pip commands outside of virtual environments.

It is generally best practice to use virtual environments with pip, so
let's migrate distros that currently use pip to use virtual environments
as well.

Because BitBake is a Python project and requires a few Python packages
that we currently install with distro package managers, if we were to
ignore system packages when creating the virtual environment the user
would either be able to build the docs or run BitBake. Let's include the
system packages in the virtual env so that isn't an issue.

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
 documentation/tools/containerfiles/Containerfile.dnf    | 6 ++++++
 documentation/tools/containerfiles/Containerfile.zypper | 7 +++++++
 documentation/tools/host_packages_scripts/pip3_docs.sh  | 4 +++-
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/documentation/tools/containerfiles/Containerfile.dnf b/documentation/tools/containerfiles/Containerfile.dnf
index fd47fde34..01e838ef9 100644
--- a/documentation/tools/containerfiles/Containerfile.dnf
+++ b/documentation/tools/containerfiles/Containerfile.dnf
@@ -9,6 +9,12 @@ ARG PIP3=pip3_docs.sh
 
 ENV TEXTOOLDIR=/opt/docs-build-tex-tools
 ENV PATH="$PATH:$TEXTOOLDIR/tl/bin/x86_64-linux"
+# RUN . /tmp/yocto-venv/bin/activate doesn't work because env variables set in
+# RUN don't persist, so set up the virtualenv manually.
+# Unsetting env variables in Containerfile is difficult, so we hope it works
+# fine without doing that for PYTHONHOME (see activate script content).
+ENV VIRTUAL_ENV=/tmp/yocto-venv
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
 
 # relative to the location of the dockerfile
 COPY --chmod=777 ${ESSENTIAL} /temp/host_packages_essential.sh
diff --git a/documentation/tools/containerfiles/Containerfile.zypper b/documentation/tools/containerfiles/Containerfile.zypper
index 3850b9ff9..204185efd 100644
--- a/documentation/tools/containerfiles/Containerfile.zypper
+++ b/documentation/tools/containerfiles/Containerfile.zypper
@@ -13,6 +13,13 @@ COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
 COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
 COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
 
+# RUN . /tmp/yocto-venv/bin/activate doesn't work because env variables set in
+# RUN don't persist, so set up the virtualenv manually.
+# Unsetting env variables in Containerfile is difficult, so we hope it works
+# fine without doing that for PYTHONHOME (see activate script content).
+ENV VIRTUAL_ENV=/tmp/yocto-venv
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
 # Zypper doesn't have environment variables to specify whether to run in
 # non-interactive mode like Debian does with DEBIAN_FRONTEND and piping yes to
 # the scripts doesn't need to be enough as well, so let's force all zypper calls
diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
index 907ecec55..9aa7f16fb 100644
--- a/documentation/tools/host_packages_scripts/pip3_docs.sh
+++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
@@ -1 +1,3 @@
-sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
+python3 -m venv --clear --system-site-packages /tmp/yocto-venv
+. /tmp/yocto-venv/bin/activate
+python3 -m pip install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton

-- 
2.53.0

[PATCH v4 2/3] consistently use pip packages for all distros

[Quentin Schulz]

From: Quentin Schulz <quentin.schulz@cherry.de>

Until now, Debian and Ubuntu had special instructions for installing
Python packages to build the docs. Specifically, the packages that all
other distros are installing via pip were installed from the package
feed (apt).

In the next commit, a new Python module will be required, which isn't
available in the package feed. We thus have two options, have a pip3
Debian/Ubuntu-specific set of instructions to install this one package,
or migrate packages currently installed through the package manager to
use pip like other distros. To lower maintenance burden and the
possibility of missing to update both "generic" and "Debian/Ubuntu"
variants of pip instructions, the latter was chosen.

Suggested-by: Antonin Godard <antonin.godard@bootlin.com>
Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
 documentation/ref-manual/system-requirements.rst         |  3 +++
 documentation/tools/build-docs-container                 |  4 +++-
 documentation/tools/containerfiles/Containerfile.apt     | 10 ++++++++++
 documentation/tools/host_packages_scripts/ubuntu_docs.sh |  2 +-
 4 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst
index f9f32f2a8..e7c1c3916 100644
--- a/documentation/ref-manual/system-requirements.rst
+++ b/documentation/ref-manual/system-requirements.rst
@@ -171,6 +171,9 @@ Here are the packages needed to build Project documentation manuals:
 .. literalinclude:: ../tools/host_packages_scripts/ubuntu_docs.sh
    :language: shell
 
+.. literalinclude:: ../tools/host_packages_scripts/pip3_docs.sh
+   :language: shell
+
 In addition to the previous packages, here are the packages needed to build the
 documentation in PDF format:
 
diff --git a/documentation/tools/build-docs-container b/documentation/tools/build-docs-container
index 23c3dfe33..f2f54044c 100755
--- a/documentation/tools/build-docs-container
+++ b/documentation/tools/build-docs-container
@@ -115,6 +115,7 @@ main ()
       essential=ubuntu_essential.sh
       docs=ubuntu_docs.sh
       docs_pdf=ubuntu_docs_pdf.sh
+      pip3=pip3_docs.sh
       ;;
     "fedora:39"*|\
     "fedora:40"*|\
@@ -151,6 +152,7 @@ main ()
       essential=ubuntu_essential.sh
       docs=ubuntu_docs.sh
       docs_pdf=ubuntu_docs_pdf.sh
+      pip3=pip3_docs.sh
       ;;
     *)
       echo "$image not supported!"
@@ -166,7 +168,7 @@ main ()
     --build-arg ESSENTIAL="host_packages_scripts/$essential" \
     --build-arg DOCS="host_packages_scripts/$docs" \
     --build-arg DOCS_PDF="host_packages_scripts/$docs_pdf" \
-    --build-arg PIP3="host_packages_scripts/${pip3:-}" \
+    --build-arg PIP3="host_packages_scripts/$pip3" \
     --file "$SCRIPT_DIR/containerfiles/$containerfile" \
     "$SCRIPT_DIR"
 
diff --git a/documentation/tools/containerfiles/Containerfile.apt b/documentation/tools/containerfiles/Containerfile.apt
index a573786f0..13cbc791f 100644
--- a/documentation/tools/containerfiles/Containerfile.apt
+++ b/documentation/tools/containerfiles/Containerfile.apt
@@ -5,6 +5,7 @@ ARG INCLUDE_ESSENTIAL_PACKAGES=0
 ARG ESSENTIAL=ubuntu_essential.sh
 ARG DOCS=ubuntu_docs.sh
 ARG DOCS_PDF=ubuntu_docs_pdf.sh
+ARG PIP3=pip3_docs.sh
 
 ENV DEBIAN_FRONTEND=noninteractive
 ARG TZ=Europe/Vienna
@@ -13,6 +14,14 @@ ARG TZ=Europe/Vienna
 COPY --chmod=777 ${ESSENTIAL} /temp/host_packages_essential.sh
 COPY --chmod=777 ${DOCS} /temp/host_packages_docs.sh
 COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
+COPY --chmod=777 ${PIP3} /temp/pip3_docs.sh
+
+# RUN . /tmp/yocto-venv/bin/activate doesn't work because env variables set in
+# RUN don't persist, so set up the virtualenv manually.
+# Unsetting env variables in Containerfile is difficult, so we hope it works
+# fine without doing that for PYTHONHOME (see activate script content).
+ENV VIRTUAL_ENV=/tmp/yocto-venv
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
 
 RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
  && apt-get update \
@@ -20,6 +29,7 @@ RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
  && if [ "$INCLUDE_ESSENTIAL_PACKAGES" = "1" ]; then yes | /temp/host_packages_essential.sh; fi \
  && yes | /temp/host_packages_docs.sh \
  && yes | /temp/host_packages_docs_pdf.sh \
+ && yes | /temp/pip3_docs.sh \
  && apt-get --yes autoremove \
  && apt-get clean \
  && rm -rf /temp
diff --git a/documentation/tools/host_packages_scripts/ubuntu_docs.sh b/documentation/tools/host_packages_scripts/ubuntu_docs.sh
index c322b61e4..db0743021 100644
--- a/documentation/tools/host_packages_scripts/ubuntu_docs.sh
+++ b/documentation/tools/host_packages_scripts/ubuntu_docs.sh
@@ -1 +1 @@
-sudo apt install git librsvg2-bin locales make python3-saneyaml python3-sphinx-rtd-theme python3-sphinx-copybutton sphinx
+sudo apt install git librsvg2-bin locales make python3-saneyaml python3-pip python3-venv

-- 
2.53.0

[PATCH v4 3/3] convert SVGs to PDF and PNG using sphinxcontrib.rsvgconverter plugin

[Quentin Schulz]

From: Quentin Schulz <quentin.schulz@cherry.de>

The sphinxcontrib.rsvgconverter plugin allows to generate a PDF or PNG
from an SVG.
This is what we already do manually via Make targets but it isn't good
enough.

Sphinx generates a cache upon first parsing and stores it in a doctrees
directory. Sphinx claims that it can be shared between all builders[1].
For glob patterns in image or figure directives, Sphinx will look for
all possible matches in the source tree and store those in a
"candidates" map for each file, along with the associated mimetype. When
building, Sphinx will then look in this map to try to find a match in
the current builder's supported_image_types. If none is found, the build
will fail.

The latexpdf (using the LaTeXBuilder) target does not support SVGs by
default[2]. We used Make to generate PDFs from them before generating
the doc PDF though (see PDFs variable and %.pdf in Makefile) and that
type is supported by default[2].

The epub (via the Epub3Builder) target does support SVGs by default[3]
but we disabled their support in commit ff3876ca4910 ("conf.py: use PNG
first in EPUB output"). We used Make to generate PNGs from them before
generating the doc epub though (see PNGs variable and %.png in Makefile)
and that type is supported (c.f. Epub3Builder.supported_image_types in
our conf.py).

The issue is that this is done transparently from Sphinx. When we
generate the PDFs or PNGs variants of the SVGs, we put them in-tree
directly along their source file. Then, when caching, Sphinx will find
both the source file and the appropriate variant. However, the cache
isn't updated if there are new files in the tree and the source rST
files aren't modified. So, the cache will not have its map updated and
we won't be able to find the new variant when building for a
non-SVG-compatible builder. Take the following scenario:

- start from a clean source file (fresh clone or git clean -ffdx)
- build the html target (which supports SVGs by default[4])
- sphinx will find all the files in the source tree matching the glob
  pattern in ".. image:: test.*", in our case only an SVG since the PDFs
  and PNGs are only generated for the latexpdf and epub targets
  respectively. The cache will only store the path to the SVG file
  because it is the only source file that matches the glob,
- attempt to build the epub target (which doesn't support SVGs, only
  PNGs)
- Sphinx checks for the file to include for '.. image:: test.*' and
  finds an SVG in the cache map and then check the list of supported
  image types for the Epub3Builder and find that it doesn't support
  that. It cannot find anything satisfying the dependency and thus fails
  to build.

This scenario can easily be reproduced by running the `make all` command
since the html builder will be used first, then epub and finally
latexpdf.

The `make publish` target works by chance, because the epub builder is
built first and will cache SVG + PNG for each glob, then the latexpdf
builder is built and supports PNGs so that's fine and then html, which
supports SVG as well.

To fix this issue, we could simply always generate PDFs and PNGs of all
SVGs in the source tree, but this isn't ideal.

Instead, let's use an ImageConverter from a Sphinx plugin. This allows
to map a plugin as being able to generate a file of type Y from a file
of type X. When Sphinx wants to build an image, it'll try to find the
image with the type the current builder supports in the cache. If it
cannot, it's going to try to find an ImageConverter plugin that is able
to convert one of the image types in cache with one of the image types
the current builder supports. Then Sphinx will call this plugin to
generate the file and put it into the build directory (not in the
source!).

This allows to simplify the Makefile as well and is a much cleaner
approach.

The epub target is removed as the catch-all target contains the same
instructions as the epub target we remove in this commit.

sphinxcontrib.rsvgconverter is a plugin from
sphinxcontrib-svg2pdfconverter python module. SVGs to PNGs is only
supported since 2.0.0.

[1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html#cmdoption-sphinx-build-d
[2] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.latex.LaTeXBuilder.supported_image_types
[3] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.epub3.Epub3Builder.supported_image_types
[4] https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.html.StandaloneHTMLBuilder

Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
---
 documentation/Makefile                             | 27 +++-------------------
 documentation/Pipfile                              |  2 ++
 documentation/conf.py                              |  1 +
 .../tools/host_packages_scripts/pip3_docs.sh       |  2 +-
 4 files changed, 7 insertions(+), 25 deletions(-)

diff --git a/documentation/Makefile b/documentation/Makefile
index e144a50b4..897db2b5a 100644
--- a/documentation/Makefile
+++ b/documentation/Makefile
@@ -10,11 +10,8 @@ VALEOPTS       ?= --no-wrap --glob '!migration-guides/release-notes-*.rst'
 SOURCEDIR      = .
 VALEDOCS       ?= $(SOURCEDIR)
 SPHINXLINTDOCS ?= $(SOURCEDIR)
-IMAGEDIRS      = */svg
 BUILDDIR       = _build
 DESTDIR        = final
-SVG2PNG        = rsvg-convert
-SVG2PDF        = rsvg-convert
 
 ifeq ($(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi),0)
 $(error "The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed")
@@ -24,7 +21,7 @@ endif
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: all checks help Makefile clean stylecheck publish epub latexpdf
+.PHONY: all checks help Makefile clean stylecheck publish latexpdf
 
 publish: Makefile checks epub latexpdf html singlehtml
 	rm -rf $(BUILDDIR)/$(DESTDIR)/
@@ -35,22 +32,8 @@ publish: Makefile checks epub latexpdf html singlehtml
 	cp $(BUILDDIR)/singlehtml/index.html $(BUILDDIR)/$(DESTDIR)/singleindex.html
 	sed -i -e 's@index.html#@singleindex.html#@g' $(BUILDDIR)/$(DESTDIR)/singleindex.html
 
-# Build a list of SVG files to convert to PDFs
-PDFs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.pdf,$(wildcard $(SOURCEDIR)/$(dir)/*.svg)))
-
-# Build a list of SVG files to convert to PNGs
-PNGs := $(foreach dir, $(IMAGEDIRS), $(patsubst %.svg,%.png,$(wildcard $(SOURCEDIR)/$(dir)/*.svg)))
-
-# Pattern rule for converting SVG to PDF
-%.pdf : %.svg
-	$(SVG2PDF) --format=pdf --output=$@ $<
-
-# Pattern rule for converting SVG to PNG
-%.png : %.svg
-	$(SVG2PNG) --format=png --output=$@ $<
-
 clean:
-	@rm -rf $(BUILDDIR) $(PNGs) $(PDFs) poky.yaml sphinx-static/switchers.js releases.rst
+	@rm -rf $(BUILDDIR) poky.yaml sphinx-static/switchers.js releases.rst
 
 checks:
 	$(SOURCEDIR)/tools/check-glossaries --docs-dir $(SOURCEDIR)
@@ -62,14 +45,10 @@ stylecheck:
 sphinx-lint:
 	sphinx-lint $(SPHINXLINTDOCS)
 
-epub: $(PNGs)
-	$(SOURCEDIR)/set_versions.py
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
-
 # Note: we need to pass buf_size here (which is also configurable from
 # texmf.cnf), to avoid following error:
 #   Unable to read an entire line---bufsize=200000. Please increase buf_size in texmf.cnf.
-latexpdf: $(PDFs)
+latexpdf:
 	$(SOURCEDIR)/set_versions.py
 	buf_size=10000000 $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
diff --git a/documentation/Pipfile b/documentation/Pipfile
index 1fa2df31e..67fce078d 100644
--- a/documentation/Pipfile
+++ b/documentation/Pipfile
@@ -10,6 +10,8 @@ sphinx = "*"
 sphinx-rtd-theme = "*"
 pyyaml = "*"
 sphinx-copybutton = "*"
+# SVG to PNG only supported since 2.0.0
+sphinxcontrib-svg2pdfconverter = ">=2.0.0"
 
 [requires]
 python_version = "3"
diff --git a/documentation/conf.py b/documentation/conf.py
index 3d04f9057..5a69977cf 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -67,6 +67,7 @@ extensions = [
     'sphinx.ext.extlinks',
     'sphinx.ext.intersphinx',
     'sphinx_copybutton',
+    'sphinxcontrib.rsvgconverter',
     'yocto-vars'
 ]
 autosectionlabel_prefix_document = True
diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
index 9aa7f16fb..2c9330d70 100644
--- a/documentation/tools/host_packages_scripts/pip3_docs.sh
+++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
@@ -1,3 +1,3 @@
 python3 -m venv --clear --system-site-packages /tmp/yocto-venv
 . /tmp/yocto-venv/bin/activate
-python3 -m pip install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
+python3 -m pip install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton 'sphinxcontrib-svg2pdfconverter>=2.0.0'

-- 
2.53.0

Re: [PATCH v4 0/3] migrate Debian/Ubuntu to pip packages for docs + fix epub and latexpdf targets not finding glob images

[Antonin Godard]

Hi,

I have an issue after applying the series:

  $ ./documentation/tools/build-docs-container almalinux:9 latexpdf
  [...]
  latexmk -pdf -dvi- -ps-  'theyoctoproject.tex'
  make[1]: latexmk: No such file or directory
  make[1]: *** [Makefile:28: theyoctoproject.pdf] Error 127
  make[1]: Leaving directory '/docs/documentation/_build/latex'
  make: *** [Makefile:53: latexpdf] Error 2
  make: Leaving directory '/docs/documentation'

You reported an error with latexpdf on IRC, was that it?

Antonin

Re: [docs] [PATCH v4 1/3] use a venv for installing packages with pip

[Antonin Godard]

Hi,

On Mon Feb 16, 2026 at 12:26 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
[...]
> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
> index 907ecec55..9aa7f16fb 100644
> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
> @@ -1 +1,3 @@
> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
> +python3 -m venv --clear --system-site-packages /tmp/yocto-venv

I find /tmp a bit odd for installing a virtual environment (at least, from a
user perspective, as I can understand why you did this for the container).

Maybe you can add a comment above this line saying:

# change "/tmp/yocto-venv" to "./yocto-venv" if you want a persistent virtual environment

?

Also maybe, s/yocto-venv/yocto-docs-venv/?

Antonin

Re: [docs] [PATCH v4 1/3] use a venv for installing packages with pip

[Quentin Schulz]

Hi Antonin,

On 2/18/26 9:18 AM, Antonin Godard wrote:
> Hi,
> 
> On Mon Feb 16, 2026 at 12:26 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
> [...]
>> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
>> index 907ecec55..9aa7f16fb 100644
>> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
>> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
>> @@ -1 +1,3 @@
>> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
>> +python3 -m venv --clear --system-site-packages /tmp/yocto-venv
> 
> I find /tmp a bit odd for installing a virtual environment (at least, from a
> user perspective, as I can understand why you did this for the container).
> 
> Maybe you can add a comment above this line saying:
> 
> # change "/tmp/yocto-venv" to "./yocto-venv" if you want a persistent virtual environment
> 
> ?
> 

I can do yet another sed in Containerfile to avoid telling the user to 
do that and just have the path different in the container (see what we 
do for zypper install commands). I tried mounting the volume 
(yocto-docs) as an overlay within the container (:O in podman) but that 
didn't work, so we have to have the venv outside of the mounting point.

> Also maybe, s/yocto-venv/yocto-docs-venv/?
> 

I was about to interject that because we use --system-site-packages, we 
could still run bitbake from within the venv and thus renaming it is 
unnecessary, but then I tried building the docs locally on my f43 system 
and this series doesn't actually work.

It complains about not being able to find sphinxcontrib.rsvgconverter 
plugin while it definitely is installed!

Then I remembered having a somewhat similar issue in my Python 
pet-project when I had the same package installed twice, once 
system-wide and once in a venv... Turns out that a package which 
installs a command/script doesn't seem to work with venv (at least with 
my sample pool of... 2). It would always use the system's when using the 
command (and it'd be fine with python3 -m <package> as far as I remember).
sphinx-build actually is a script, c.f. 
https://github.com/sphinx-doc/sphinx/blob/master/pyproject.toml#L99. So 
when our Makefile calls sphinx-build, it may very well be calling the 
system package if there's one and not the command for the sphinx within 
the venv. This results in not being able to find the modules installed 
in the venv (that is, sphinxcontrib.rsvgconverter). But Quentin, you 
would say, simply replace sphinx-build with python3 -m sphinx and all 
shall be well! Now the sphinx Python module used is indeed the one from 
the venv, but it cannot find the system modules anymore. If anyone has 
any idea what's going on, let me know but I don't think I'll investigate 
this further.

Instead, I'll NOT pass --system-site-packages to venv such that we only 
have packages from PyPI (which means, not being able to run BitBake from 
that venv). Now... this still doesn't resolve the issue of sphinx-build 
calling the module from the system if it exists... So I plan to change 
sphinx-build to python3 -m sphinx in our Makefile to work-around that. 
Any objection? Did I miss something or get something wrong maybe?

Cheers,
Quentin

Re: [docs] [PATCH v4 1/3] use a venv for installing packages with pip

[Antonin Godard]

Hi,

On Mon Feb 23, 2026 at 6:33 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 2/18/26 9:18 AM, Antonin Godard wrote:
>> Hi,
>> 
>> On Mon Feb 16, 2026 at 12:26 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
>> [...]
>>> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> index 907ecec55..9aa7f16fb 100644
>>> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> @@ -1 +1,3 @@
>>> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
>>> +python3 -m venv --clear --system-site-packages /tmp/yocto-venv
>> 
>> I find /tmp a bit odd for installing a virtual environment (at least, from a
>> user perspective, as I can understand why you did this for the container).
>> 
>> Maybe you can add a comment above this line saying:
>> 
>> # change "/tmp/yocto-venv" to "./yocto-venv" if you want a persistent virtual environment
>> 
>> ?
>> 
>
> I can do yet another sed in Containerfile to avoid telling the user to 
> do that and just have the path different in the container (see what we 
> do for zypper install commands). I tried mounting the volume 
> (yocto-docs) as an overlay within the container (:O in podman) but that 

You mean yocto-venv?

> didn't work, so we have to have the venv outside of the mounting point.

Sure, I guess it's best if we can show the user the proper way of setting things
up while doing it our way.

>> Also maybe, s/yocto-venv/yocto-docs-venv/?
>> 
>
> I was about to interject that because we use --system-site-packages, we 
> could still run bitbake from within the venv and thus renaming it is 
> unnecessary, but then I tried building the docs locally on my f43 system 
> and this series doesn't actually work.
>
> It complains about not being able to find sphinxcontrib.rsvgconverter 
> plugin while it definitely is installed!
>
> Then I remembered having a somewhat similar issue in my Python 
> pet-project when I had the same package installed twice, once 
> system-wide and once in a venv... Turns out that a package which 
> installs a command/script doesn't seem to work with venv (at least with 
> my sample pool of... 2). It would always use the system's when using the 
> command (and it'd be fine with python3 -m <package> as far as I remember).
> sphinx-build actually is a script, c.f. 
> https://github.com/sphinx-doc/sphinx/blob/master/pyproject.toml#L99. So 
> when our Makefile calls sphinx-build, it may very well be calling the 
> system package if there's one and not the command for the sphinx within 
> the venv. This results in not being able to find the modules installed 
> in the venv (that is, sphinxcontrib.rsvgconverter). But Quentin, you 
> would say, simply replace sphinx-build with python3 -m sphinx and all 
> shall be well! Now the sphinx Python module used is indeed the one from 
> the venv, but it cannot find the system modules anymore. If anyone has 
> any idea what's going on, let me know but I don't think I'll investigate 
> this further.
>
> Instead, I'll NOT pass --system-site-packages to venv such that we only 
> have packages from PyPI (which means, not being able to run BitBake from 
> that venv). Now... this still doesn't resolve the issue of sphinx-build 
> calling the module from the system if it exists...

I did some tests:

➜  ~ mktemp -d
/tmp/tmp.M4eSbciQhZ
➜  ~ cd /tmp/tmp.M4eSbciQhZ
➜  which sphinx-build
/usr/bin/sphinx-build
➜  python -m venv .venv
➜  source .venv/bin/activate
➜  pip install sphinx >/dev/null
➜  which sphinx-build
/usr/bin/sphinx-build

Maybe you're also hitting this. Now after running:

➜  rehash
➜  which sphinx-build
/tmp/tmp.M4eSbciQhZ/.venv/bin/sphinx-build

Looks like it's working now. rehash seems specific to zsh in my case, bash does
not know this command. But bash does not need to rehash, it works out of the box
for me with bash.

> So I plan to change sphinx-build to python3 -m sphinx in our Makefile to
> work-around that. Any objection? Did I miss something or get something wrong
> maybe?

I thought of doing:

diff --git a/documentation/tools/containerfiles/Containerfile.apt b/documentation/tools/containerfiles/Containerfile.apt
index a573786f0..0861354fb 100644
--- a/documentation/tools/containerfiles/Containerfile.apt
+++ b/documentation/tools/containerfiles/Containerfile.apt
@@ -17,9 +17,9 @@ COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
 RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
  && apt-get update \
  && apt-get install -y sudo \
- && if [ "$INCLUDE_ESSENTIAL_PACKAGES" = "1" ]; then yes | /temp/host_packages_essential.sh; fi \
- && yes | /temp/host_packages_docs.sh \
- && yes | /temp/host_packages_docs_pdf.sh \
+ && if [ "$INCLUDE_ESSENTIAL_PACKAGES" = "1" ]; then yes | bash /temp/host_packages_essential.sh; fi \
+ && yes | bash /temp/host_packages_docs.sh \
+ && yes | bash /temp/host_packages_docs_pdf.sh \
  && apt-get --yes autoremove \
  && apt-get clean \
  && rm -rf /temp

Or add a shebang to the snippets… but that makes them non "copy-pasteable".

So, yes, I vote for "python -m sphinx" too! Seems like a good solution to me :)

Antonin

Re: [docs] [PATCH v4 1/3] use a venv for installing packages with pip

[Quentin Schulz]

Hi Antonin,

On 2/24/26 9:28 AM, Antonin Godard wrote:
> Hi,
> 
> On Mon Feb 23, 2026 at 6:33 PM CET, Quentin Schulz wrote:
>> Hi Antonin,
>>
>> On 2/18/26 9:18 AM, Antonin Godard wrote:
>>> Hi,
>>>
>>> On Mon Feb 16, 2026 at 12:26 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
>>> [...]
>>>> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>>> index 907ecec55..9aa7f16fb 100644
>>>> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
>>>> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>>> @@ -1 +1,3 @@
>>>> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
>>>> +python3 -m venv --clear --system-site-packages /tmp/yocto-venv
>>>
>>> I find /tmp a bit odd for installing a virtual environment (at least, from a
>>> user perspective, as I can understand why you did this for the container).
>>>
>>> Maybe you can add a comment above this line saying:
>>>
>>> # change "/tmp/yocto-venv" to "./yocto-venv" if you want a persistent virtual environment
>>>
>>> ?
>>>
>>
>> I can do yet another sed in Containerfile to avoid telling the user to
>> do that and just have the path different in the container (see what we
>> do for zypper install commands). I tried mounting the volume
>> (yocto-docs) as an overlay within the container (:O in podman) but that
> 
> You mean yocto-venv?
> 

No, I meant yocto-docs git repo.

The venv is within the container rootfs (set up as part of the 
Containerfile instructions).

We mount the yocto-docs repo into /docs in the container with a volume 
mount, see the declaration of args_run array in tools/build-docs-container.

We need to have a way to have the container /docs (where yocto-docs-venv 
venv is) overlayed by a volume mount such that the yocto-docs git repo 
and the venv appear in /docs within the container. I am not sure this is 
possible (at least, my short experiment with :O in volume mount 
option[1] didn't succeed. I think this is rather used as "make the 
content added by the container runtime overlay the volume mount from the 
host such that when you exits the container the overlayed stuff is 
removed". It seems you can tell podman to use a different upperdir and 
workdir but my understanding is that this is for storing the overlayed 
stuff outside of the container such that it persists for later container 
runtimes).

[1] 
https://docs.podman.io/en/latest/markdown/podman-run.1.html#volume-v-source-volume-host-dir-container-dir-options 
Overlay Volume Mounts

>> didn't work, so we have to have the venv outside of the mounting point.
> 
> Sure, I guess it's best if we can show the user the proper way of setting things
> up while doing it our way.
> 
>>> Also maybe, s/yocto-venv/yocto-docs-venv/?
>>>
>>
>> I was about to interject that because we use --system-site-packages, we
>> could still run bitbake from within the venv and thus renaming it is
>> unnecessary, but then I tried building the docs locally on my f43 system
>> and this series doesn't actually work.
>>
>> It complains about not being able to find sphinxcontrib.rsvgconverter
>> plugin while it definitely is installed!
>>
>> Then I remembered having a somewhat similar issue in my Python
>> pet-project when I had the same package installed twice, once
>> system-wide and once in a venv... Turns out that a package which
>> installs a command/script doesn't seem to work with venv (at least with
>> my sample pool of... 2). It would always use the system's when using the
>> command (and it'd be fine with python3 -m <package> as far as I remember).
>> sphinx-build actually is a script, c.f.
>> https://github.com/sphinx-doc/sphinx/blob/master/pyproject.toml#L99. So
>> when our Makefile calls sphinx-build, it may very well be calling the
>> system package if there's one and not the command for the sphinx within
>> the venv. This results in not being able to find the modules installed
>> in the venv (that is, sphinxcontrib.rsvgconverter). But Quentin, you
>> would say, simply replace sphinx-build with python3 -m sphinx and all
>> shall be well! Now the sphinx Python module used is indeed the one from
>> the venv, but it cannot find the system modules anymore. If anyone has
>> any idea what's going on, let me know but I don't think I'll investigate
>> this further.
>>
>> Instead, I'll NOT pass --system-site-packages to venv such that we only
>> have packages from PyPI (which means, not being able to run BitBake from
>> that venv). Now... this still doesn't resolve the issue of sphinx-build
>> calling the module from the system if it exists...
> 
> I did some tests:
> 
> ➜  ~ mktemp -d
> /tmp/tmp.M4eSbciQhZ
> ➜  ~ cd /tmp/tmp.M4eSbciQhZ
> ➜  which sphinx-build
> /usr/bin/sphinx-build
> ➜  python -m venv .venv
> ➜  source .venv/bin/activate
> ➜  pip install sphinx >/dev/null
> ➜  which sphinx-build
> /usr/bin/sphinx-build
> 
> Maybe you're also hitting this. Now after running:
> 
Nope, works perfectly fine on my side. Returns what it should, that is, 
sphinx-build from the venv. However, *running* the command (e.g. 
sphinx-build --version) shows it's using the host system's and not venv's.

> ➜  rehash

hash -r is run as part of the activate script of the venv when sourced 
(see last line in the script). I'm assuming this is similar to zsh's 
rehash since this script is compatible with zsh and bash[2].

I'm assuming the issue is we need to run it again after installing 
packages through pip...

Another option is to source the venv activate script again after having 
run pip commands. Or run pip without having sourced that activate 
script, e.g. like

python3 -m venv --clear --system-site-packages ./yocto-docs-venv/
./yocto-docs-venv/bin/python3 -m pip install xxx
. ./yocto-docs-venv/bin/activate

or

python3 -m venv --clear --system-site-packages ./yocto-docs-venv/
. ./yocto-docs-venv/bin/activate
python3 -m pip install xxx
. ./yocto-docs-venv/bin/activate

or we use python3 -m sphinx in Makefile. What would you prefer?

[2] https://docs.python.org/3/library/venv.html#how-venvs-work

> ➜  which sphinx-build
> /tmp/tmp.M4eSbciQhZ/.venv/bin/sphinx-build
> 
> Looks like it's working now. rehash seems specific to zsh in my case, bash does
> not know this command. But bash does not need to rehash, it works out of the box
> for me with bash.
> 

I bet it doesn't if you don't run rehash. Can you run sphinx-build 
--version, python3 -m sphinx --version and python3 -m pip list | grep 
Sphinx?

For me, the former returns the same version inside and outside the venv, 
that is, the host system's version. The second returns what's inside the 
venv, as reported by the latter.

So... I researched this a bit more and I think I may have an idea what's 
happening. sphinxcontrib is a namespace package. Many packages install 
in that namespace, e.g. sphinxcontrib-jquery but also 
sphinxcontrib-svg2pdfconverter, you know the package I'm trying to add 
in this series as a dependency?

See 
https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages 
for how to create native namespace packages.

And you know what namespace supports? Different locations per "portion" 
of the namespace, see 
https://peps.python.org/pep-0420/#differences-between-namespace-packages-and-regular-packages. 
This means you can import sphinxcontrib packages from different 
locations (host system's or venv, if you use --system-site-package for 
python3 -m venv). And you know what's important for regular packages 
that are portions of a namespace? To not break the namespace package, 
that is, NOT have a sphinxcontrib/__init__.py file... Which, lo and 
behold, seems to be what sphinxcontrib-svg2pdfconverter is doing. I'm 
assuming because it is NOT using the sphinxcontrib namespace package, 
but defining a regular package sphinxcontrib with multiple modules. This 
means that importing sphinxcontrib when you have 
sphinxcontrib-svg2pdfconverter anywhere in your host system or venv 
environment, you break multi-location support for packages in the 
namespace. Why this didn't break more than simply not being able to use 
multi-location support is beyond my understanding.

To check paths checked by the NamespaceLoader[3], you can create a venv 
with --system-site-packages, then after sourcing the venv activate 
script, run

python3 -c 'import sphinxcontrib; print(sphinxcontrib.__path__)'

It'll print a few paths that will be looked up when doing an

import sphinxcontrib.something

Now install sphinxcontrib-svg2pdfconverter and run the same command and 
you get only one path.

So... I'll be working on fixing sphinxcontrib-svg2pdfconverter and 
hopefully we don't have to wait too long for a v2.0.1 (or whatever the 
next version will be). This means we'll be able to use 
--system-site-packages again, which I think is neat.

[3] 
https://docs.python.org/3/library/importlib.html#importlib.machinery.NamespaceLoader

>> So I plan to change sphinx-build to python3 -m sphinx in our Makefile to
>> work-around that. Any objection? Did I miss something or get something wrong
>> maybe?
> 
> I thought of doing:
> 
> diff --git a/documentation/tools/containerfiles/Containerfile.apt b/documentation/tools/containerfiles/Containerfile.apt
> index a573786f0..0861354fb 100644
> --- a/documentation/tools/containerfiles/Containerfile.apt
> +++ b/documentation/tools/containerfiles/Containerfile.apt
> @@ -17,9 +17,9 @@ COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
>   RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
>    && apt-get update \
>    && apt-get install -y sudo \
> - && if [ "$INCLUDE_ESSENTIAL_PACKAGES" = "1" ]; then yes | /temp/host_packages_essential.sh; fi \
> - && yes | /temp/host_packages_docs.sh \
> - && yes | /temp/host_packages_docs_pdf.sh \
> + && if [ "$INCLUDE_ESSENTIAL_PACKAGES" = "1" ]; then yes | bash /temp/host_packages_essential.sh; fi \
> + && yes | bash /temp/host_packages_docs.sh \
> + && yes | bash /temp/host_packages_docs_pdf.sh \
>    && apt-get --yes autoremove \
>    && apt-get clean \
>    && rm -rf /temp
> 

What does this help with exactly?

> Or add a shebang to the snippets… but that makes them non "copy-pasteable".
> 

Why not, it's just a regular comment when pasted into the CLI?

Cheers,
Quentin

Re: [docs] [PATCH v4 1/3] use a venv for installing packages with pip

[Quentin Schulz]

Hi Antonin,

On 2/24/26 1:52 PM, Quentin Schulz via lists.yoctoproject.org wrote:
> Hi Antonin,
> 
> On 2/24/26 9:28 AM, Antonin Godard wrote:
>> Hi,
>>
>> On Mon Feb 23, 2026 at 6:33 PM CET, Quentin Schulz wrote:
>>> Hi Antonin,
>>>
>>> On 2/18/26 9:18 AM, Antonin Godard wrote:
>>>> Hi,
>>>>
>>>> On Mon Feb 16, 2026 at 12:26 PM CET, Quentin Schulz via 
>>>> lists.yoctoproject.org wrote:

[...]

> So... I'll be working on fixing sphinxcontrib-svg2pdfconverter and 
> hopefully we don't have to wait too long for a v2.0.1 (or whatever the 
> next version will be). This means we'll be able to use --system-site- 
> packages again, which I think is neat.
> 

See 
https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter/pull/34

Do we want to wait for this to be merged before having a look at this 
series again? This prevents us from using --system-site-packages, but we 
could decide to not have it and force all Python packages for building 
the docs to be taken only from the venv (which is what happens when 
--system-site-packages is *NOT* passed to venv at creation time).

What do you think?

Cheers,
Quentin

Re: [docs] [PATCH v4 1/3] use a venv for installing packages with pip

[Antonin Godard]

Hi,

On Tue Feb 24, 2026 at 2:55 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 2/24/26 1:52 PM, Quentin Schulz via lists.yoctoproject.org wrote:
>> Hi Antonin,
>> 
>> On 2/24/26 9:28 AM, Antonin Godard wrote:
>>> Hi,
>>>
>>> On Mon Feb 23, 2026 at 6:33 PM CET, Quentin Schulz wrote:
>>>> Hi Antonin,
>>>>
>>>> On 2/18/26 9:18 AM, Antonin Godard wrote:
>>>>> Hi,
>>>>>
>>>>> On Mon Feb 16, 2026 at 12:26 PM CET, Quentin Schulz via 
>>>>> lists.yoctoproject.org wrote:
>
> [...]
>
>> So... I'll be working on fixing sphinxcontrib-svg2pdfconverter and 
>> hopefully we don't have to wait too long for a v2.0.1 (or whatever the 
>> next version will be). This means we'll be able to use --system-site- 
>> packages again, which I think is neat.
>> 
>
> See 
> https://github.com/missinglinkelectronics/sphinxcontrib-svg2pdfconverter/pull/34
>
> Do we want to wait for this to be merged before having a look at this 
> series again? This prevents us from using --system-site-packages, but we 
> could decide to not have it and force all Python packages for building 
> the docs to be taken only from the venv (which is what happens when 
> --system-site-packages is *NOT* passed to venv at creation time).
>
> What do you think?

Let's go for python3 -m sphinx in the Makefile and no --system-site-packages
package option for the time being, and improve later if we can.

Thanks for taking the time to dig!
Antonin

Re: [PATCH v4 0/3] migrate Debian/Ubuntu to pip packages for docs + fix epub and latexpdf targets not finding glob images

[Antonin Godard]

Applied, thanks!

[1/3] use a venv for installing packages with pip
      commit: 16599c4f65d451c47104e7604b68a0658d25ac48
[2/3] consistently use pip packages for all distros
      commit: 4fbae014790373ec2c662325e54cf7f07f507dbc
[3/3] convert SVGs to PDF and PNG using sphinxcontrib.rsvgconverter plugin
      commit: ba351b35480d9c06c870a6f34d8bdbc1856b9511