Re: [docs] [PATCH v4 1/3] use a venv for installing packages with pip

["Antonin Godard" <antonin.godard@bootlin.com>]

Hi,

On Mon Feb 23, 2026 at 6:33 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 2/18/26 9:18 AM, Antonin Godard wrote:
>> Hi,
>> 
>> On Mon Feb 16, 2026 at 12:26 PM CET, Quentin Schulz via lists.yoctoproject.org wrote:
>> [...]
>>> diff --git a/documentation/tools/host_packages_scripts/pip3_docs.sh b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> index 907ecec55..9aa7f16fb 100644
>>> --- a/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> +++ b/documentation/tools/host_packages_scripts/pip3_docs.sh
>>> @@ -1 +1,3 @@
>>> -sudo pip3 install sphinx sphinx_rtd_theme pyyaml sphinx-copybutton
>>> +python3 -m venv --clear --system-site-packages /tmp/yocto-venv
>> 
>> I find /tmp a bit odd for installing a virtual environment (at least, from a
>> user perspective, as I can understand why you did this for the container).
>> 
>> Maybe you can add a comment above this line saying:
>> 
>> # change "/tmp/yocto-venv" to "./yocto-venv" if you want a persistent virtual environment
>> 
>> ?
>> 
>
> I can do yet another sed in Containerfile to avoid telling the user to 
> do that and just have the path different in the container (see what we 
> do for zypper install commands). I tried mounting the volume 
> (yocto-docs) as an overlay within the container (:O in podman) but that 

You mean yocto-venv?

> didn't work, so we have to have the venv outside of the mounting point.

Sure, I guess it's best if we can show the user the proper way of setting things
up while doing it our way.

>> Also maybe, s/yocto-venv/yocto-docs-venv/?
>> 
>
> I was about to interject that because we use --system-site-packages, we 
> could still run bitbake from within the venv and thus renaming it is 
> unnecessary, but then I tried building the docs locally on my f43 system 
> and this series doesn't actually work.
>
> It complains about not being able to find sphinxcontrib.rsvgconverter 
> plugin while it definitely is installed!
>
> Then I remembered having a somewhat similar issue in my Python 
> pet-project when I had the same package installed twice, once 
> system-wide and once in a venv... Turns out that a package which 
> installs a command/script doesn't seem to work with venv (at least with 
> my sample pool of... 2). It would always use the system's when using the 
> command (and it'd be fine with python3 -m <package> as far as I remember).
> sphinx-build actually is a script, c.f. 
> https://github.com/sphinx-doc/sphinx/blob/master/pyproject.toml#L99. So 
> when our Makefile calls sphinx-build, it may very well be calling the 
> system package if there's one and not the command for the sphinx within 
> the venv. This results in not being able to find the modules installed 
> in the venv (that is, sphinxcontrib.rsvgconverter). But Quentin, you 
> would say, simply replace sphinx-build with python3 -m sphinx and all 
> shall be well! Now the sphinx Python module used is indeed the one from 
> the venv, but it cannot find the system modules anymore. If anyone has 
> any idea what's going on, let me know but I don't think I'll investigate 
> this further.
>
> Instead, I'll NOT pass --system-site-packages to venv such that we only 
> have packages from PyPI (which means, not being able to run BitBake from 
> that venv). Now... this still doesn't resolve the issue of sphinx-build 
> calling the module from the system if it exists...

I did some tests:

➜  ~ mktemp -d
/tmp/tmp.M4eSbciQhZ
➜  ~ cd /tmp/tmp.M4eSbciQhZ
➜  which sphinx-build
/usr/bin/sphinx-build
➜  python -m venv .venv
➜  source .venv/bin/activate
➜  pip install sphinx >/dev/null
➜  which sphinx-build
/usr/bin/sphinx-build

Maybe you're also hitting this. Now after running:

➜  rehash
➜  which sphinx-build
/tmp/tmp.M4eSbciQhZ/.venv/bin/sphinx-build

Looks like it's working now. rehash seems specific to zsh in my case, bash does
not know this command. But bash does not need to rehash, it works out of the box
for me with bash.

> So I plan to change sphinx-build to python3 -m sphinx in our Makefile to
> work-around that. Any objection? Did I miss something or get something wrong
> maybe?

I thought of doing:

diff --git a/documentation/tools/containerfiles/Containerfile.apt b/documentation/tools/containerfiles/Containerfile.apt
index a573786f0..0861354fb 100644
--- a/documentation/tools/containerfiles/Containerfile.apt
+++ b/documentation/tools/containerfiles/Containerfile.apt
@@ -17,9 +17,9 @@ COPY --chmod=777 ${DOCS_PDF} /temp/host_packages_docs_pdf.sh
 RUN ln -fs "/usr/share/zoneinfo/$TZ" /etc/localtime \
  && apt-get update \
  && apt-get install -y sudo \
- && if [ "$INCLUDE_ESSENTIAL_PACKAGES" = "1" ]; then yes | /temp/host_packages_essential.sh; fi \
- && yes | /temp/host_packages_docs.sh \
- && yes | /temp/host_packages_docs_pdf.sh \
+ && if [ "$INCLUDE_ESSENTIAL_PACKAGES" = "1" ]; then yes | bash /temp/host_packages_essential.sh; fi \
+ && yes | bash /temp/host_packages_docs.sh \
+ && yes | bash /temp/host_packages_docs_pdf.sh \
  && apt-get --yes autoremove \
  && apt-get clean \
  && rm -rf /temp

Or add a shebang to the snippets… but that makes them non "copy-pasteable".

So, yes, I vote for "python -m sphinx" too! Seems like a good solution to me :)

Antonin