Re: [PATCH v6 00/21] Split sphinx call logic from docs Makefile

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Em Tue, 16 Sep 2025 12:22:36 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Hi Jon,
> 
> This series should probably be called:
> 
>     "Move the trick-or-treat build hacks accumulated over time
>      into a single place and document them."
> 
> as this reflects its main goal. As such:
> 
> - it places the jobserver logic on a library;
> - it removes sphinx/parallel-wrapper.sh;
> - the code now properly implements a jobserver-aware logic
>   to do the parallelism when called via GNU make, failing back to
>   "-j" when there's  no jobserver;
> - converts check-variable-fonts.sh to Python and uses it via
>   function call;
> - drops an extra script to generate man pages, adding a makefile
>   target for it;
> - ensures that return code is 0 when PDF successfully builds;
> - about half of the script is comments and documentation.
> 
> I tried to do my best to document all tricks that are inside the
> script. This way, the docs build steps is now documented.
> 
> It should be noticed that it is out of the scope of this series
> to change the implementation. Surely the process can be improved,
> but first let's consolidate and document everything on a single
> place.
> 
> Such script was written in a way that it can be called either
> directly or via a Makefile. Running outside Makefile is
> interesting specially when debug is needed. The command line
> interface replaces the need of having lots of env vars before
> calling sphinx-build:
> 
>     $ ./tools/docs/sphinx-build-wrapper --help
>     usage: sphinx-build-wrapper [-h]
>            [--sphinxdirs SPHINXDIRS [SPHINXDIRS ...]] [--conf CONF]
>            [--builddir BUILDDIR] [--theme THEME] [--css CSS] [--paper {,a4,letter}] [-v]
>            [-j JOBS] [-i] [-V [VENV]]
>            {cleandocs,linkcheckdocs,htmldocs,epubdocs,texinfodocs,infodocs,mandocs,latexdocs,pdfdocs,xmldocs}
> 
>     Kernel documentation builder
> 
>     positional arguments:
>       {cleandocs,linkcheckdocs,htmldocs,epubdocs,texinfodocs,infodocs,mandocs,latexdocs,pdfdocs,xmldocs}
>                             Documentation target to build
> 
>     options:
>       -h, --help            show this help message and exit
>       --sphinxdirs SPHINXDIRS [SPHINXDIRS ...]
>                             Specific directories to build
>       --conf CONF           Sphinx configuration file
>       --builddir BUILDDIR   Sphinx configuration file
>       --theme THEME         Sphinx theme to use
>       --css CSS             Custom CSS file for HTML/EPUB
>       --paper {,a4,letter}  Paper size for LaTeX/PDF output
>       -v, --verbose         place build in verbose mode
>       -j, --jobs JOBS       Sets number of jobs to use with sphinx-build
>       -i, --interactive     Change latex default to run in interactive mode
>       -V, --venv [VENV]     If used, run Sphinx from a venv dir (default dir: sphinx_latest)
> 
> the only mandatory argument is the target, which is identical with
> "make" targets.
> 
> The call inside Makefile doesn't use the last four arguments. They're
> there to help identifying problems at the build:
> 
>     -v makes the output verbose;
>     -j helps to test parallelism;
>     -i runs latexmk in interactive mode, allowing to debug PDF
>        build issues;
>     -V is useful when testing it with different venvs.
> 
> When used with GNU make (or some other make which implements jobserver),
> a call like:
> 
>     make -j <targets> htmldocs
> 
> will make the wrapper to automatically use POSIX jobserver to claim 
> the number of available job slots, calling sphinx-build with a
> "-j" parameter reflecting it. ON such case, the default can be
> overriden via SPHINXDIRS argument.
> 
> Visiable changes when compared with the old behavior:

Visiable -> Visible

> When V=0, the only visible difference is that:
> - pdfdocs target now returns 0 on success, 1 on failures.
>   This addresses an issue over the current process where we
>   it always return success even on failures;
> - it will now print the name of PDF files that failed to build,
>   if any.
> 
> In verbose mode, sphinx-build-wrapper and sphinx-build command lines
> are now displayed.

It will also print a summary of all PDF built files, for pdfdocs
target in verbose mode.

> 
> ---
> 
> v6:
> - On success, PDF output is identical as before when V=0;
> - when V=1 is used, PDF output will print a build summary,
>   as on v5;
> - solved a problem when multiple PDF files have the same
>   basename but are located on different directories;
> - merged a patch series converting check-variable-fonts.sh
>   to Python. Its logic is now called directly without running
>   a subprocess.
> - venv patch moved to the end.
> 
> v5:
> - merged comments with the script;
> - placed n_jobs on a separate function;
> - nitpick: dropped a for loop used instead of list append.
> 
> v4:
> - updated references for sphinx-pre-install after its rename;
> - added some extra patches to add more options to python_version,
>   allowing it to bail out and suggest alternatives;
> - added a patch at the end to explicitly break doc builds when
>   python3 points to python3.6 or older.
> 
> v3:
> - rebased on the top of docs-next;
> - added two patches to build man files that were on a separate
>   patch series.
> 
> v2:
> - there's no generic exception handler anymore;
> - it moves sphinx-pre-install to tools/docs;
> - the logic which ensures a minimal Python version got moved
>   to a library, which is now used by both pre-install and wrapper;
> - The first wrapper (05/13) doesn't contain comments (except for
>   shebang and SPDX). The goal is to help showing the size increase
>   when moving from Makefile to Python. Some file increase is
>   unavoidable, as Makefile is more compact: no includes, multple
>   statements per line, no argparse, etc;
> - The second patch adds docstrings and comments. It has almost
>   the same size of the code itself;
> - I moved the venv logic to a third wrapper patch;
> - I fixed an issue at the paraller build logic;
> - There are no generic except blocks anymore.
> 
> 
> Mauro Carvalho Chehab (21):
>   scripts/jobserver-exec: move the code to a class
>   scripts/jobserver-exec: move its class to the lib directory
>   scripts/jobserver-exec: add a help message
>   scripts: check-variable-fonts.sh: convert to Python
>   tools/docs: check-variable-fonts.py: split into a lib and an exec file
>   scripts: sphinx-pre-install: move it to tools/docs
>   tools/docs: python_version: move version check from sphinx-pre-install
>   tools/docs: python_version: drop a debug print
>   tools/docs: python_version: allow check for alternatives and bail out
>   tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
>   docs: parallel-wrapper.sh: remove script
>   docs: Makefile: document latex/PDF PAPER= parameter
>   tools/docs: sphinx-build-wrapper: add an argument for LaTeX
>     interactive mode
>   tools/docs,scripts: sphinx-*: prevent sphinx-build crashes
>   tools/docs: sphinx-build-wrapper: allow building PDF files in parallel
>   tools/docs: sphinx-build-wrapper: Fix output for duplicated names
>   docs: add support to build manpages from kerneldoc output
>   tools: kernel-doc: add a see also section at man pages
>   scripts: kdoc_parser.py: warn about Python version only once
>   tools/docs: sphinx-* break documentation bulds on openSUSE
>   tools/docs: sphinx-build-wrapper: add support to run inside venv
> 
>  Documentation/Makefile                        | 136 +--
>  Documentation/doc-guide/kernel-doc.rst        |  29 +-
>  Documentation/doc-guide/sphinx.rst            |   4 +-
>  Documentation/sphinx/kerneldoc-preamble.sty   |   2 +-
>  Documentation/sphinx/parallel-wrapper.sh      |  33 -
>  .../translations/it_IT/doc-guide/sphinx.rst   |   4 +-
>  .../translations/zh_CN/doc-guide/sphinx.rst   |   4 +-
>  Documentation/translations/zh_CN/how-to.rst   |   2 +-
>  MAINTAINERS                                   |   4 +-
>  Makefile                                      |   2 +-
>  scripts/check-variable-fonts.sh               | 115 ---
>  scripts/jobserver-exec                        |  88 +-
>  scripts/lib/jobserver.py                      | 149 ++++
>  scripts/lib/kdoc/kdoc_files.py                |   5 +-
>  scripts/lib/kdoc/kdoc_output.py               |  84 +-
>  scripts/lib/kdoc/kdoc_parser.py               |   7 +-
>  scripts/split-man.pl                          |  28 -
>  tools/docs/check-variable-fonts.py            |  23 +
>  tools/docs/lib/latex_fonts.py                 | 162 ++++
>  tools/docs/lib/python_version.py              | 178 ++++
>  tools/docs/sphinx-build-wrapper               | 791 ++++++++++++++++++
>  {scripts => tools/docs}/sphinx-pre-install    | 135 +--
>  22 files changed, 1502 insertions(+), 483 deletions(-)
>  delete mode 100644 Documentation/sphinx/parallel-wrapper.sh
>  delete mode 100755 scripts/check-variable-fonts.sh
>  create mode 100755 scripts/lib/jobserver.py
>  delete mode 100755 scripts/split-man.pl
>  create mode 100755 tools/docs/check-variable-fonts.py
>  create mode 100755 tools/docs/lib/latex_fonts.py
>  create mode 100644 tools/docs/lib/python_version.py
>  create mode 100755 tools/docs/sphinx-build-wrapper
>  rename {scripts => tools/docs}/sphinx-pre-install (93%)
> 



Thanks,
Mauro