[PATCH 0/6] Some improvements for the doc build system

[PATCH 0/6] Some improvements for the doc build system

[Mauro Carvalho Chehab]

Hi Jon,

This series contain some patches from my parser-yaml one that
aren't directly related to it. It basically addresses some issues
at the build system. It also adds a script that I wrote with the
purpose of checking backward problems when building against
older toolchains.

IMO, the best is to merge and apply it before the YAML series.

I'll be respining the YAML later, with some additional changes.

Regards,
Mauro

Mauro Carvalho Chehab (6):
  docs: conf.py: properly handle include and exclude patterns
  docs: Makefile: disable check rules on make cleandocs
  scripts: scripts/test_doc_build.py: add script to test doc build
  scripts/test_doc_build.py: make capture assynchronous
  scripts: test_doc_build.py: better control its output
  docs: sphinx: add a file with the requirements for lowest version

 Documentation/Makefile                    |   2 +
 Documentation/conf.py                     |  67 +++-
 Documentation/doc-guide/sphinx.rst        |  23 ++
 Documentation/sphinx/min_requirements.txt |  10 +
 scripts/test_doc_build.py                 | 382 ++++++++++++++++++++++
 5 files changed, 480 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/sphinx/min_requirements.txt
 create mode 100755 scripts/test_doc_build.py

-- 
2.49.0

[PATCH 1/6] docs: conf.py: properly handle include and exclude patterns

[Mauro Carvalho Chehab]

When one does:
	make SPHINXDIRS="foo" htmldocs

All patterns would be relative to Documentation/foo, which
causes the include/exclude patterns like:

	include_patterns = [
		...
		f'foo/*.{ext}',
	]

to break. This is not what it is expected. Address it by
adding a logic to dynamically adjust the pattern when
SPHINXDIRS is used.

That allows adding parsers for other file types.

It should be noticed that include_patterns was added on
Sphinx 5.1:
	https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-include_patterns

So, a backward-compatible code is needed when we start
using it for real.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/conf.py | 67 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 63 insertions(+), 4 deletions(-)

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 12de52a2b17e..4ba4ee45e599 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -17,6 +17,66 @@ import os
 import sphinx
 import shutil
 
+# Get Sphinx version
+major, minor, patch = sphinx.version_info[:3]
+
+# Include_patterns were added on Sphinx 5.1
+if (major < 5) or (major == 5 and minor < 1):
+    has_include_patterns = False
+else:
+    has_include_patterns = True
+    # Include patterns that don't contain directory names, in glob format
+    include_patterns = ['**.rst']
+
+# Location of Documentation/ directory
+doctree = os.path.abspath('.')
+
+# Exclude of patterns that don't contain directory names, in glob format.
+exclude_patterns = []
+
+# List of patterns that contain directory names in glob format.
+dyn_include_patterns = []
+dyn_exclude_patterns = ['output']
+
+# Properly handle include/exclude patterns
+# ----------------------------------------
+
+def update_patterns(app):
+
+    """
+    On Sphinx, all directories are relative to what it is passed as
+    SOURCEDIR parameter for sphinx-build. Due to that, all patterns
+    that have directory names on it need to be dynamically set, after
+    converting them to a relative patch.
+
+    As Sphinx doesn't include any patterns outside SOURCEDIR, we should
+    exclude relative patterns that start with "../".
+    """
+
+    sourcedir = app.srcdir  # full path to the source directory
+    builddir = os.environ.get("BUILDDIR")
+
+    # setup include_patterns dynamically
+    if has_include_patterns:
+        for p in dyn_include_patterns:
+            full = os.path.join(doctree, p)
+
+            rel_path = os.path.relpath(full, start = app.srcdir)
+            if rel_path.startswith("../"):
+                continue
+
+            app.config.include_patterns.append(rel_path)
+
+    # setup exclude_patterns dynamically
+    for p in dyn_exclude_patterns:
+        full = os.path.join(doctree, p)
+
+        rel_path = os.path.relpath(full, start = app.srcdir)
+        if rel_path.startswith("../"):
+            continue
+
+        app.config.exclude_patterns.append(rel_path)
+
 # helper
 # ------
 
@@ -219,10 +279,6 @@ language = 'en'
 # Else, today_fmt is used as the format for a strftime call.
 #today_fmt = '%B %d, %Y'
 
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-exclude_patterns = ['output']
-
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
 #default_role = None
@@ -516,3 +572,6 @@ kerneldoc_srctree = '..'
 # the last statement in the conf.py file
 # ------------------------------------------------------------------------------
 loadConfig(globals())
+
+def setup(app):
+	app.connect('builder-inited', update_patterns)
-- 
2.49.0

[PATCH 2/6] docs: Makefile: disable check rules on make cleandocs

[Mauro Carvalho Chehab]

It doesn't make sense to check for missing ABI and documents
when cleaning the tree.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Reviewed-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index d30d66ddf1ad..b98477df5ddf 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -5,6 +5,7 @@
 # for cleaning
 subdir- := devicetree/bindings
 
+ifneq ($(MAKECMDGOALS),cleandocs)
 # Check for broken documentation file references
 ifeq ($(CONFIG_WARN_MISSING_DOCUMENTS),y)
 $(shell $(srctree)/scripts/documentation-file-ref-check --warn)
@@ -14,6 +15,7 @@ endif
 ifeq ($(CONFIG_WARN_ABI_ERRORS),y)
 $(shell $(srctree)/scripts/get_abi.py --dir $(srctree)/Documentation/ABI validate)
 endif
+endif
 
 # You can set these variables from the command line.
 SPHINXBUILD   = sphinx-build
-- 
2.49.0

[PATCH 3/6] scripts: scripts/test_doc_build.py: add script to test doc build

[Mauro Carvalho Chehab]

Testing Sphinx backward-compatibility is hard, as per version
minimal Python dependency requirements can be a nightmare.

Add a script to help automate such checks.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/test_doc_build.py | 241 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 241 insertions(+)
 create mode 100755 scripts/test_doc_build.py

diff --git a/scripts/test_doc_build.py b/scripts/test_doc_build.py
new file mode 100755
index 000000000000..482716fbe91d
--- /dev/null
+++ b/scripts/test_doc_build.py
@@ -0,0 +1,241 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+# Copyright(c) 2025: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
+#
+# pylint: disable=C0103,R1715
+
+"""
+Install minimal supported requirements for different Sphinx versions
+and optionally test the build.
+"""
+
+import argparse
+import os.path
+import sys
+import time
+
+from subprocess import run
+
+# Minimal python version supported by the building system
+python_bin = "python3.9"
+
+# Starting from 8.0.2, Python 3.9 becomes too old
+python_changes = {(8, 0, 2): "python3"}
+
+# Sphinx versions to be installed and their incremental requirements
+sphinx_requirements = {
+    (3, 4, 3): {
+        "alabaster": "0.7.13",
+        "babel": "2.17.0",
+        "certifi": "2025.6.15",
+        "charset-normalizer": "3.4.2",
+        "docutils": "0.15",
+        "idna": "3.10",
+        "imagesize": "1.4.1",
+        "Jinja2": "3.0.3",
+        "MarkupSafe": "2.0",
+        "packaging": "25.0",
+        "Pygments": "2.19.1",
+        "PyYAML": "5.1",
+        "requests": "2.32.4",
+        "snowballstemmer": "3.0.1",
+        "sphinxcontrib-applehelp": "1.0.4",
+        "sphinxcontrib-devhelp": "1.0.2",
+        "sphinxcontrib-htmlhelp": "2.0.1",
+        "sphinxcontrib-jsmath": "1.0.1",
+        "sphinxcontrib-qthelp": "1.0.3",
+        "sphinxcontrib-serializinghtml": "1.1.5",
+        "urllib3": "2.4.0",
+    },
+    (3, 5, 4): {},
+    (4, 0, 3): {
+        "docutils": "0.17.1",
+        "PyYAML": "5.1",
+    },
+    (4, 1, 2): {},
+    (4, 3, 2): {},
+    (4, 4, 0): {},
+    (4, 5, 0): {},
+    (5, 0, 2): {},
+    (5, 1, 1): {},
+    (5, 2, 3): {
+        "Jinja2": "3.1.2",
+        "MarkupSafe": "2.0",
+        "PyYAML": "5.3.1",
+    },
+    (5, 3, 0): {
+        "docutils": "0.18.1",
+        "PyYAML": "5.3.1",
+    },
+    (6, 0, 1): {},
+    (6, 1, 3): {},
+    (6, 2, 1): {
+        "PyYAML": "5.4.1",
+    },
+    (7, 0, 1): {},
+    (7, 1, 2): {},
+    (7, 2, 3): {
+        "PyYAML": "6.0.1",
+        "sphinxcontrib-serializinghtml": "1.1.9",
+    },
+    (7, 3, 7): {
+        "alabaster": "0.7.14",
+        "PyYAML": "6.0.1",
+    },
+    (7, 4, 7): {
+        "docutils": "0.20",
+        "PyYAML": "6.0.1",
+    },
+    (8, 0, 2): {},
+    (8, 1, 3): {
+        "PyYAML": "6.0.1",
+        "sphinxcontrib-applehelp": "1.0.7",
+        "sphinxcontrib-devhelp": "1.0.6",
+        "sphinxcontrib-htmlhelp": "2.0.6",
+        "sphinxcontrib-qthelp": "1.0.6",
+    },
+    (8, 2, 3): {
+        "PyYAML": "6.0.1",
+        "sphinxcontrib-serializinghtml": "1.1.9",
+    },
+}
+
+
+def parse_version(ver_str):
+    """Convert a version string into a tuple."""
+
+    return tuple(map(int, ver_str.split(".")))
+
+
+parser = argparse.ArgumentParser(description="Build docs for different sphinx_versions.")
+
+parser.add_argument('-v', '--version', help='Sphinx single version',
+                    type=parse_version)
+parser.add_argument('--min-version', "--min", help='Sphinx minimal version',
+                    type=parse_version)
+parser.add_argument('--max-version', "--max", help='Sphinx maximum version',
+                    type=parse_version)
+parser.add_argument('-a', '--make_args',
+                    help='extra arguments for make htmldocs, like SPHINXDIRS=netlink/specs',
+                    nargs="*")
+parser.add_argument('-w', '--write', help='write a requirements.txt file',
+                    action='store_true')
+parser.add_argument('-m', '--make',
+                    help='Make documentation',
+                    action='store_true')
+parser.add_argument('-i', '--wait-input',
+                    help='Wait for an enter before going to the next version',
+                    action='store_true')
+
+args = parser.parse_args()
+
+if not args.make_args:
+    args.make_args = []
+
+if args.version:
+    if args.min_version or args.max_version:
+        sys.exit("Use either --version or --min-version/--max-version")
+    else:
+        args.min_version = args.version
+        args.max_version = args.version
+
+sphinx_versions = sorted(list(sphinx_requirements.keys()))
+
+if not args.min_version:
+    args.min_version = sphinx_versions[0]
+
+if not args.max_version:
+    args.max_version = sphinx_versions[-1]
+
+first_run = True
+cur_requirements = {}
+built_time = {}
+
+for cur_ver, new_reqs in sphinx_requirements.items():
+    cur_requirements.update(new_reqs)
+
+    if cur_ver in python_changes:
+        python_bin = python_changes[cur_ver]
+
+    ver = ".".join(map(str, cur_ver))
+
+    if args.min_version:
+        if cur_ver < args.min_version:
+            continue
+
+    if args.max_version:
+        if cur_ver > args.max_version:
+            break
+
+    if not first_run and args.wait_input and args.make:
+        ret = input("Press Enter to continue or 'a' to abort: ").strip().lower()
+        if ret == "a":
+            print("Aborted.")
+            sys.exit()
+    else:
+        first_run = False
+
+    venv_dir = f"Sphinx_{ver}"
+    req_file = f"requirements_{ver}.txt"
+
+    print(f"\nSphinx {ver} with {python_bin}")
+
+    # Create venv
+    run([python_bin, "-m", "venv", venv_dir], check=True)
+    pip = os.path.join(venv_dir, "bin/pip")
+
+    # Create install list
+    reqs = []
+    for pkg, verstr in cur_requirements.items():
+        reqs.append(f"{pkg}=={verstr}")
+
+    reqs.append(f"Sphinx=={ver}")
+
+    run([pip, "install"] + reqs, check=True)
+
+    # Freeze environment
+    result = run([pip, "freeze"], capture_output=True, text=True, check=True)
+
+    # Pip install succeeded. Write requirements file
+    if args.write:
+        with open(req_file, "w", encoding="utf-8") as fp:
+            fp.write(result.stdout)
+
+    if args.make:
+        start_time = time.time()
+
+        # Prepare a venv environment
+        env = os.environ.copy()
+        bin_dir = os.path.join(venv_dir, "bin")
+        env["PATH"] = bin_dir + ":" + env["PATH"]
+        env["VIRTUAL_ENV"] = venv_dir
+        if "PYTHONHOME" in env:
+            del env["PYTHONHOME"]
+
+        # Test doc build
+        run(["make", "cleandocs"], env=env, check=True)
+        make = ["make"] + args.make_args + ["htmldocs"]
+
+        print(f". {bin_dir}/activate")
+        print(" ".join(make))
+        print("deactivate")
+        run(make, env=env, check=True)
+
+        end_time = time.time()
+        elapsed_time = end_time - start_time
+        hours, minutes = divmod(elapsed_time, 3600)
+        minutes, seconds = divmod(minutes, 60)
+
+        hours = int(hours)
+        minutes = int(minutes)
+        seconds = int(seconds)
+
+        built_time[ver] = f"{hours:02d}:{minutes:02d}:{seconds:02d}"
+
+        print(f"Finished doc build for Sphinx {ver}. Elapsed time: {built_time[ver]}")
+
+if args.make:
+    print()
+    print("Summary:")
+    for ver, elapsed_time in sorted(built_time.items()):
+        print(f"\tSphinx {ver} elapsed time: {elapsed_time}")
-- 
2.49.0

[PATCH 4/6] scripts/test_doc_build.py: make capture assynchronous

[Mauro Carvalho Chehab]

Prepare the tool to allow writing the output into log files.
For such purpose, receive stdin/stdout messages asynchronously.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/test_doc_build.py | 393 +++++++++++++++++++++++++-------------
 1 file changed, 255 insertions(+), 138 deletions(-)

diff --git a/scripts/test_doc_build.py b/scripts/test_doc_build.py
index 482716fbe91d..94f2f2d8c3b7 100755
--- a/scripts/test_doc_build.py
+++ b/scripts/test_doc_build.py
@@ -2,7 +2,7 @@
 # SPDX-License-Identifier: GPL-2.0
 # Copyright(c) 2025: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
 #
-# pylint: disable=C0103,R1715
+# pylint: disable=R0903,R0913,R0914,R0917
 
 """
 Install minimal supported requirements for different Sphinx versions
@@ -10,20 +10,20 @@ and optionally test the build.
 """
 
 import argparse
+import asyncio
 import os.path
 import sys
 import time
-
-from subprocess import run
+import subprocess
 
 # Minimal python version supported by the building system
-python_bin = "python3.9"
+MINIMAL_PYTHON_VERSION = "python3.9"
 
 # Starting from 8.0.2, Python 3.9 becomes too old
-python_changes = {(8, 0, 2): "python3"}
+PYTHON_VER_CHANGES = {(8, 0, 2): "python3"}
 
 # Sphinx versions to be installed and their incremental requirements
-sphinx_requirements = {
+SPHINX_REQUIREMENTS = {
     (3, 4, 3): {
         "alabaster": "0.7.13",
         "babel": "2.17.0",
@@ -101,141 +101,258 @@ sphinx_requirements = {
 }
 
 
+class AsyncCommands:
+    """Excecute command synchronously"""
+
+    stdout = None
+    stderr = None
+    output = None
+
+    async def _read(self, stream, verbose, is_info):
+        """Ancillary routine to capture while displaying"""
+
+        while stream is not None:
+            line = await stream.readline()
+            if line:
+                out = line.decode("utf-8", errors="backslashreplace")
+                self.output += out
+                if is_info:
+                    if verbose:
+                        print(out.rstrip("\n"))
+
+                    self.stdout += out
+                else:
+                    if verbose:
+                        print(out.rstrip("\n"), file=sys.stderr)
+
+                    self.stderr += out
+            else:
+                break
+
+    async def run(self, cmd, capture_output=False, check=False,
+                  env=None, verbose=True):
+
+        """
+        Execute an arbitrary command, handling errors.
+
+        Please notice that this class is not thread safe
+        """
+
+        self.stdout = ""
+        self.stderr = ""
+        self.output = ""
+
+        if verbose:
+            print("$ ", " ".join(cmd))
+
+        proc = await asyncio.create_subprocess_exec(cmd[0],
+                                                    *cmd[1:],
+                                                    env=env,
+                                                    stdout=asyncio.subprocess.PIPE,
+                                                    stderr=asyncio.subprocess.PIPE)
+
+        # Handle input and output in realtime
+        await asyncio.gather(
+            self._read(proc.stdout, verbose, True),
+            self._read(proc.stderr, verbose, False),
+        )
+
+        await proc.wait()
+
+        if check and proc.returncode > 0:
+            raise subprocess.CalledProcessError(returncode=proc.returncode,
+                                                cmd=" ".join(cmd),
+                                                output=self.stdout,
+                                                stderr=self.stderr)
+
+        if capture_output:
+            if proc.returncode > 0:
+                print("Error {proc.returncode}", file=sys.stderr)
+                return ""
+
+            return self.output
+
+        ret = subprocess.CompletedProcess(args=cmd,
+                                          returncode=proc.returncode,
+                                          stdout=self.stdout,
+                                          stderr=self.stderr)
+
+        return ret
+
+
+class SphinxVenv:
+    """
+    Installs Sphinx on one virtual env per Sphinx version with a minimal
+    set of dependencies, adjusting them to each specific version.
+    """
+
+    def __init__(self):
+        """Initialize instance variables"""
+
+        self.built_time = {}
+        self.first_run = True
+
+    async def _handle_version(self, args, cur_ver, cur_requirements, python_bin):
+        """Handle a single Sphinx version"""
+
+        cmd = AsyncCommands()
+
+        ver = ".".join(map(str, cur_ver))
+
+        if not self.first_run and args.wait_input and args.make:
+            ret = input("Press Enter to continue or 'a' to abort: ").strip().lower()
+            if ret == "a":
+                print("Aborted.")
+                sys.exit()
+        else:
+            self.first_run = False
+
+        venv_dir = f"Sphinx_{ver}"
+        req_file = f"requirements_{ver}.txt"
+
+        print(f"\nSphinx {ver} with {python_bin}")
+
+        # Create venv
+        await cmd.run([python_bin, "-m", "venv", venv_dir], check=True)
+        pip = os.path.join(venv_dir, "bin/pip")
+
+        # Create install list
+        reqs = []
+        for pkg, verstr in cur_requirements.items():
+            reqs.append(f"{pkg}=={verstr}")
+
+        reqs.append(f"Sphinx=={ver}")
+
+        await cmd.run([pip, "install"] + reqs, check=True, verbose=True)
+
+        # Freeze environment
+        result = await cmd.run([pip, "freeze"], verbose=False, check=True)
+
+        # Pip install succeeded. Write requirements file
+        if args.write:
+            with open(req_file, "w", encoding="utf-8") as fp:
+                fp.write(result.stdout)
+
+        if args.make:
+            start_time = time.time()
+
+            # Prepare a venv environment
+            env = os.environ.copy()
+            bin_dir = os.path.join(venv_dir, "bin")
+            env["PATH"] = bin_dir + ":" + env["PATH"]
+            env["VIRTUAL_ENV"] = venv_dir
+            if "PYTHONHOME" in env:
+                del env["PYTHONHOME"]
+
+            # Test doc build
+            await cmd.run(["make", "cleandocs"], env=env, check=True)
+            make = ["make"] + args.make_args + ["htmldocs"]
+
+            print(f". {bin_dir}/activate")
+            print(" ".join(make))
+            print("deactivate")
+            await cmd.run(make, env=env, check=True)
+
+            end_time = time.time()
+            elapsed_time = end_time - start_time
+            hours, minutes = divmod(elapsed_time, 3600)
+            minutes, seconds = divmod(minutes, 60)
+
+            hours = int(hours)
+            minutes = int(minutes)
+            seconds = int(seconds)
+
+            self.built_time[ver] = f"{hours:02d}:{minutes:02d}:{seconds:02d}"
+
+            print(f"Finished doc build for Sphinx {ver}. Elapsed time: {self.built_time[ver]}")
+
+    async def run(self, args):
+        """
+        Navigate though multiple Sphinx versions, handling each of them
+        on a loop.
+        """
+
+        cur_requirements = {}
+        python_bin = MINIMAL_PYTHON_VERSION
+
+        for cur_ver, new_reqs in SPHINX_REQUIREMENTS.items():
+            cur_requirements.update(new_reqs)
+
+            if cur_ver in PYTHON_VER_CHANGES:          # pylint: disable=R1715
+
+                python_bin = PYTHON_VER_CHANGES[cur_ver]
+
+            if args.min_version:
+                if cur_ver < args.min_version:
+                    continue
+
+            if args.max_version:
+                if cur_ver > args.max_version:
+                    break
+
+            await self._handle_version(args, cur_ver, cur_requirements,
+                                       python_bin)
+
+        if args.make:
+            print()
+            print("Summary:")
+            for ver, elapsed_time in sorted(self.built_time.items()):
+                print(f"\tSphinx {ver} elapsed time: {elapsed_time}")
+
+
 def parse_version(ver_str):
     """Convert a version string into a tuple."""
 
     return tuple(map(int, ver_str.split(".")))
 
 
-parser = argparse.ArgumentParser(description="Build docs for different sphinx_versions.")
-
-parser.add_argument('-v', '--version', help='Sphinx single version',
-                    type=parse_version)
-parser.add_argument('--min-version', "--min", help='Sphinx minimal version',
-                    type=parse_version)
-parser.add_argument('--max-version', "--max", help='Sphinx maximum version',
-                    type=parse_version)
-parser.add_argument('-a', '--make_args',
-                    help='extra arguments for make htmldocs, like SPHINXDIRS=netlink/specs',
-                    nargs="*")
-parser.add_argument('-w', '--write', help='write a requirements.txt file',
-                    action='store_true')
-parser.add_argument('-m', '--make',
-                    help='Make documentation',
-                    action='store_true')
-parser.add_argument('-i', '--wait-input',
-                    help='Wait for an enter before going to the next version',
-                    action='store_true')
-
-args = parser.parse_args()
-
-if not args.make_args:
-    args.make_args = []
-
-if args.version:
-    if args.min_version or args.max_version:
-        sys.exit("Use either --version or --min-version/--max-version")
-    else:
-        args.min_version = args.version
-        args.max_version = args.version
-
-sphinx_versions = sorted(list(sphinx_requirements.keys()))
-
-if not args.min_version:
-    args.min_version = sphinx_versions[0]
-
-if not args.max_version:
-    args.max_version = sphinx_versions[-1]
-
-first_run = True
-cur_requirements = {}
-built_time = {}
-
-for cur_ver, new_reqs in sphinx_requirements.items():
-    cur_requirements.update(new_reqs)
-
-    if cur_ver in python_changes:
-        python_bin = python_changes[cur_ver]
-
-    ver = ".".join(map(str, cur_ver))
-
-    if args.min_version:
-        if cur_ver < args.min_version:
-            continue
-
-    if args.max_version:
-        if cur_ver > args.max_version:
-            break
-
-    if not first_run and args.wait_input and args.make:
-        ret = input("Press Enter to continue or 'a' to abort: ").strip().lower()
-        if ret == "a":
-            print("Aborted.")
-            sys.exit()
-    else:
-        first_run = False
-
-    venv_dir = f"Sphinx_{ver}"
-    req_file = f"requirements_{ver}.txt"
-
-    print(f"\nSphinx {ver} with {python_bin}")
-
-    # Create venv
-    run([python_bin, "-m", "venv", venv_dir], check=True)
-    pip = os.path.join(venv_dir, "bin/pip")
-
-    # Create install list
-    reqs = []
-    for pkg, verstr in cur_requirements.items():
-        reqs.append(f"{pkg}=={verstr}")
-
-    reqs.append(f"Sphinx=={ver}")
-
-    run([pip, "install"] + reqs, check=True)
-
-    # Freeze environment
-    result = run([pip, "freeze"], capture_output=True, text=True, check=True)
-
-    # Pip install succeeded. Write requirements file
-    if args.write:
-        with open(req_file, "w", encoding="utf-8") as fp:
-            fp.write(result.stdout)
-
-    if args.make:
-        start_time = time.time()
-
-        # Prepare a venv environment
-        env = os.environ.copy()
-        bin_dir = os.path.join(venv_dir, "bin")
-        env["PATH"] = bin_dir + ":" + env["PATH"]
-        env["VIRTUAL_ENV"] = venv_dir
-        if "PYTHONHOME" in env:
-            del env["PYTHONHOME"]
-
-        # Test doc build
-        run(["make", "cleandocs"], env=env, check=True)
-        make = ["make"] + args.make_args + ["htmldocs"]
-
-        print(f". {bin_dir}/activate")
-        print(" ".join(make))
-        print("deactivate")
-        run(make, env=env, check=True)
-
-        end_time = time.time()
-        elapsed_time = end_time - start_time
-        hours, minutes = divmod(elapsed_time, 3600)
-        minutes, seconds = divmod(minutes, 60)
-
-        hours = int(hours)
-        minutes = int(minutes)
-        seconds = int(seconds)
-
-        built_time[ver] = f"{hours:02d}:{minutes:02d}:{seconds:02d}"
-
-        print(f"Finished doc build for Sphinx {ver}. Elapsed time: {built_time[ver]}")
-
-if args.make:
-    print()
-    print("Summary:")
-    for ver, elapsed_time in sorted(built_time.items()):
-        print(f"\tSphinx {ver} elapsed time: {elapsed_time}")
+async def main():
+    """Main program"""
+
+    parser = argparse.ArgumentParser(description="Build docs for different sphinx_versions.")
+
+    parser.add_argument('-v', '--version', help='Sphinx single version',
+                        type=parse_version)
+    parser.add_argument('--min-version', "--min", help='Sphinx minimal version',
+                        type=parse_version)
+    parser.add_argument('--max-version', "--max", help='Sphinx maximum version',
+                        type=parse_version)
+    parser.add_argument('-a', '--make_args',
+                        help='extra arguments for make htmldocs, like SPHINXDIRS=netlink/specs',
+                        nargs="*")
+    parser.add_argument('-w', '--write', help='write a requirements.txt file',
+                        action='store_true')
+    parser.add_argument('-m', '--make',
+                        help='Make documentation',
+                        action='store_true')
+    parser.add_argument('-i', '--wait-input',
+                        help='Wait for an enter before going to the next version',
+                        action='store_true')
+
+    args = parser.parse_args()
+
+    if not args.make_args:
+        args.make_args = []
+
+    if args.version:
+        if args.min_version or args.max_version:
+            sys.exit("Use either --version or --min-version/--max-version")
+        else:
+            args.min_version = args.version
+            args.max_version = args.version
+
+    sphinx_versions = sorted(list(SPHINX_REQUIREMENTS.keys()))
+
+    if not args.min_version:
+        args.min_version = sphinx_versions[0]
+
+    if not args.max_version:
+        args.max_version = sphinx_versions[-1]
+
+    venv = SphinxVenv()
+    await venv.run(args)
+
+
+# Call main method
+if __name__ == "__main__":
+    asyncio.run(main())
-- 
2.49.0

[PATCH 5/6] scripts: test_doc_build.py: better control its output

[Mauro Carvalho Chehab]

Now that asyncio is supported, allow userspace to adjust
verbosity level and direct the script output to a file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/test_doc_build.py | 78 +++++++++++++++++++++++++--------------
 1 file changed, 51 insertions(+), 27 deletions(-)

diff --git a/scripts/test_doc_build.py b/scripts/test_doc_build.py
index 94f2f2d8c3b7..5b9eb2c0bf01 100755
--- a/scripts/test_doc_build.py
+++ b/scripts/test_doc_build.py
@@ -2,7 +2,7 @@
 # SPDX-License-Identifier: GPL-2.0
 # Copyright(c) 2025: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
 #
-# pylint: disable=R0903,R0913,R0914,R0917
+# pylint: disable=R0903,R0912,R0913,R0914,R0917,C0301
 
 """
 Install minimal supported requirements for different Sphinx versions
@@ -104,9 +104,22 @@ SPHINX_REQUIREMENTS = {
 class AsyncCommands:
     """Excecute command synchronously"""
 
-    stdout = None
-    stderr = None
-    output = None
+    def __init__(self, fp=None):
+
+        self.stdout = None
+        self.stderr = None
+        self.output = None
+        self.fp = fp
+
+    def log(self, out, verbose, is_info=True):
+        if verbose:
+            if is_info:
+                print(out.rstrip("\n"))
+            else:
+                print(out.rstrip("\n"), file=sys.stderr)
+
+        if self.fp:
+            self.fp.write(out.rstrip("\n") + "\n")
 
     async def _read(self, stream, verbose, is_info):
         """Ancillary routine to capture while displaying"""
@@ -115,16 +128,10 @@ class AsyncCommands:
             line = await stream.readline()
             if line:
                 out = line.decode("utf-8", errors="backslashreplace")
-                self.output += out
+                self.log(out, verbose, is_info)
                 if is_info:
-                    if verbose:
-                        print(out.rstrip("\n"))
-
                     self.stdout += out
                 else:
-                    if verbose:
-                        print(out.rstrip("\n"), file=sys.stderr)
-
                     self.stderr += out
             else:
                 break
@@ -140,10 +147,8 @@ class AsyncCommands:
 
         self.stdout = ""
         self.stderr = ""
-        self.output = ""
 
-        if verbose:
-            print("$ ", " ".join(cmd))
+        self.log("$ " + " ".join(cmd), verbose)
 
         proc = await asyncio.create_subprocess_exec(cmd[0],
                                                     *cmd[1:],
@@ -167,7 +172,7 @@ class AsyncCommands:
 
         if capture_output:
             if proc.returncode > 0:
-                print("Error {proc.returncode}", file=sys.stderr)
+                self.log(f"Error {proc.returncode}", verbose=True, is_info=False)
                 return ""
 
             return self.output
@@ -192,10 +197,11 @@ class SphinxVenv:
         self.built_time = {}
         self.first_run = True
 
-    async def _handle_version(self, args, cur_ver, cur_requirements, python_bin):
+    async def _handle_version(self, args, fp,
+                              cur_ver, cur_requirements, python_bin):
         """Handle a single Sphinx version"""
 
-        cmd = AsyncCommands()
+        cmd = AsyncCommands(fp)
 
         ver = ".".join(map(str, cur_ver))
 
@@ -210,10 +216,11 @@ class SphinxVenv:
         venv_dir = f"Sphinx_{ver}"
         req_file = f"requirements_{ver}.txt"
 
-        print(f"\nSphinx {ver} with {python_bin}")
+        cmd.log(f"\nSphinx {ver} with {python_bin}", verbose=True)
 
         # Create venv
-        await cmd.run([python_bin, "-m", "venv", venv_dir], check=True)
+        await cmd.run([python_bin, "-m", "venv", venv_dir],
+                      verbose=args.verbose, check=True)
         pip = os.path.join(venv_dir, "bin/pip")
 
         # Create install list
@@ -223,7 +230,7 @@ class SphinxVenv:
 
         reqs.append(f"Sphinx=={ver}")
 
-        await cmd.run([pip, "install"] + reqs, check=True, verbose=True)
+        await cmd.run([pip, "install"] + reqs, check=True, verbose=args.verbose)
 
         # Freeze environment
         result = await cmd.run([pip, "freeze"], verbose=False, check=True)
@@ -248,10 +255,11 @@ class SphinxVenv:
             await cmd.run(["make", "cleandocs"], env=env, check=True)
             make = ["make"] + args.make_args + ["htmldocs"]
 
-            print(f". {bin_dir}/activate")
-            print(" ".join(make))
-            print("deactivate")
-            await cmd.run(make, env=env, check=True)
+            if args.verbose:
+                print(f". {bin_dir}/activate")
+            await cmd.run(make, env=env, check=True, verbose=True)
+            if args.verbose:
+                print("deactivate")
 
             end_time = time.time()
             elapsed_time = end_time - start_time
@@ -264,7 +272,7 @@ class SphinxVenv:
 
             self.built_time[ver] = f"{hours:02d}:{minutes:02d}:{seconds:02d}"
 
-            print(f"Finished doc build for Sphinx {ver}. Elapsed time: {self.built_time[ver]}")
+            cmd.log(f"Finished doc build for Sphinx {ver}. Elapsed time: {self.built_time[ver]}", verbose=True)
 
     async def run(self, args):
         """
@@ -272,6 +280,15 @@ class SphinxVenv:
         on a loop.
         """
 
+        if args.log:
+            fp = open(args.log, "w", encoding="utf-8")
+            if not args.verbose:
+                args.verbose = False
+        else:
+            fp = None
+            if not args.verbose:
+                args.verbose = True
+
         cur_requirements = {}
         python_bin = MINIMAL_PYTHON_VERSION
 
@@ -290,7 +307,7 @@ class SphinxVenv:
                 if cur_ver > args.max_version:
                     break
 
-            await self._handle_version(args, cur_ver, cur_requirements,
+            await self._handle_version(args, fp, cur_ver, cur_requirements,
                                        python_bin)
 
         if args.make:
@@ -299,6 +316,8 @@ class SphinxVenv:
             for ver, elapsed_time in sorted(self.built_time.items()):
                 print(f"\tSphinx {ver} elapsed time: {elapsed_time}")
 
+        if fp:
+            fp.close()
 
 def parse_version(ver_str):
     """Convert a version string into a tuple."""
@@ -311,7 +330,7 @@ async def main():
 
     parser = argparse.ArgumentParser(description="Build docs for different sphinx_versions.")
 
-    parser.add_argument('-v', '--version', help='Sphinx single version',
+    parser.add_argument('-V', '--version', help='Sphinx single version',
                         type=parse_version)
     parser.add_argument('--min-version', "--min", help='Sphinx minimal version',
                         type=parse_version)
@@ -328,6 +347,11 @@ async def main():
     parser.add_argument('-i', '--wait-input',
                         help='Wait for an enter before going to the next version',
                         action='store_true')
+    parser.add_argument('-v', '--verbose',
+                        help='Verbose all commands',
+                        action='store_true')
+    parser.add_argument('-l', '--log',
+                        help='Log command output on a file')
 
     args = parser.parse_args()
 
-- 
2.49.0

[PATCH 6/6] docs: sphinx: add a file with the requirements for lowest version

[Mauro Carvalho Chehab]

Those days, it is hard to install a virtual env that would
build docs with Sphinx 3.4.3, as even python 3.13 is not
compatible anymore with it.

	/usr/bin/python3.9 -m venv sphinx_3.4.3
	. sphinx_3.4.3/bin/activate
	pip install -r Documentation/sphinx/min_requirements.txt

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/doc-guide/sphinx.rst        | 23 +++++++++++++++++++++++
 Documentation/sphinx/min_requirements.txt | 10 ++++++++++
 2 files changed, 33 insertions(+)
 create mode 100644 Documentation/sphinx/min_requirements.txt

diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 5a91df105141..607589592bfb 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -131,6 +131,29 @@ It supports two optional parameters:
 ``--no-virtualenv``
 	Use OS packaging for Sphinx instead of Python virtual environment.
 
+Installing Sphinx Minimal Version
+---------------------------------
+
+When changing Sphinx build system, it is important to ensure that
+the minimal version will still be supported. Nowadays, it is
+becoming harder to do that on modern distributions, as it is not
+possible to install with Python 3.13 and above.
+
+Testing with the lowest supported Python version as defined at
+Documentation/process/changes.rst can be done by creating
+a venv with it with, and install minimal requirements with::
+
+	/usr/bin/python3.9 -m venv sphinx_min
+	. sphinx_min/bin/activate
+	pip install -r Documentation/sphinx/min_requirements.txt
+
+A more comprehensive test can be done by using:
+
+	scripts/test_doc_build.py
+
+Such script create one Python venv per supported version,
+optionally building documentation for a range of Sphinx versions.
+
 
 Sphinx Build
 ============
diff --git a/Documentation/sphinx/min_requirements.txt b/Documentation/sphinx/min_requirements.txt
new file mode 100644
index 000000000000..52d9f27010e8
--- /dev/null
+++ b/Documentation/sphinx/min_requirements.txt
@@ -0,0 +1,10 @@
+alabaster >=0.7,<0.8
+docutils>=0.15,<0.18
+jinja2>=2.3,<3.1
+PyYAML>=5.1,<6.1
+Sphinx==3.4.3
+sphinxcontrib-applehelp==1.0.2
+sphinxcontrib-devhelp==1.0.1
+sphinxcontrib-htmlhelp==1.0.3
+sphinxcontrib-qthelp==1.0.2
+sphinxcontrib-serializinghtml==1.1.4
-- 
2.49.0

Re: [PATCH 0/6] Some improvements for the doc build system

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hi Jon,
>
> This series contain some patches from my parser-yaml one that
> aren't directly related to it. It basically addresses some issues
> at the build system. It also adds a script that I wrote with the
> purpose of checking backward problems when building against
> older toolchains.
>
> IMO, the best is to merge and apply it before the YAML series.

OK, I've applied it, but ... someday, I think the test_doc_build tool
should be properly documented and put somewhere under tools/testing.

jon

Re: [PATCH 0/6] Some improvements for the doc build system

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 13:39:09 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Hi Jon,
> >
> > This series contain some patches from my parser-yaml one that
> > aren't directly related to it. It basically addresses some issues
> > at the build system. It also adds a script that I wrote with the
> > purpose of checking backward problems when building against
> > older toolchains.
> >
> > IMO, the best is to merge and apply it before the YAML series.  
> 
> OK, I've applied it, but ... someday, I think the test_doc_build tool
> should be properly documented and put somewhere under tools/testing.

I added a better documentation for the tool at the v2.

With regards to move to tools, I'm not certain about it as I can see
advantages and disadvantages. 

Creating a new directory to have just one tool on it seems overkill
to me. Also, it is easier to type "scripts/..." than 
"tools/testing/build/..." :-)

There is another aspect: while doing conf.py and Documentation/Makefile
cleanup, I noticed that there are still lots of hacks inside them,
that are there from the early days when we adopted Sphinx. Perhaps
it could make sense to move part of the logic there to this new
build tool, which could, for instance, replace the logic inside
scripts/sphinx-pre-install and get rid of some magic at the Makefile
like the one which handles SPHINXDIRS.

So, at least for now, I would prefer to keep it under scripts.

Thanks,
Mauro

Re: [PATCH 0/6] Some improvements for the doc build system

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Em Sat, 21 Jun 2025 13:39:09 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>> OK, I've applied it, but ... someday, I think the test_doc_build tool
>> should be properly documented and put somewhere under tools/testing.
>
> I added a better documentation for the tool at the v2.
>
> With regards to move to tools, I'm not certain about it as I can see
> advantages and disadvantages. 
>
> Creating a new directory to have just one tool on it seems overkill
> to me. Also, it is easier to type "scripts/..." than 
> "tools/testing/build/..." :-)
>
> There is another aspect: while doing conf.py and Documentation/Makefile
> cleanup, I noticed that there are still lots of hacks inside them,
> that are there from the early days when we adopted Sphinx. Perhaps
> it could make sense to move part of the logic there to this new
> build tool, which could, for instance, replace the logic inside
> scripts/sphinx-pre-install and get rid of some magic at the Makefile
> like the one which handles SPHINXDIRS.
>
> So, at least for now, I would prefer to keep it under scripts.

I pretty strongly disagree ... scripts/ is a dumping ground, nobody
really knows what all that stuff there is, nobody is responsible for it.
Something under tools/ would be more evident as to its purpose and
maintainership.  We could maybe just do tools/docs/ and move things like
sphinx-pre-install there as well...

Anyway, I won't try to hold up this work based on that, but now you know
how I feel...:)

jon

Re: [PATCH 0/6] Some improvements for the doc build system

[Mauro Carvalho Chehab]

Em Sun, 22 Jun 2025 12:44:08 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Em Sat, 21 Jun 2025 13:39:09 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >  
> >> OK, I've applied it, but ... someday, I think the test_doc_build tool
> >> should be properly documented and put somewhere under tools/testing.  
> >
> > I added a better documentation for the tool at the v2.
> >
> > With regards to move to tools, I'm not certain about it as I can see
> > advantages and disadvantages. 
> >
> > Creating a new directory to have just one tool on it seems overkill
> > to me. Also, it is easier to type "scripts/..." than 
> > "tools/testing/build/..." :-)
> >
> > There is another aspect: while doing conf.py and Documentation/Makefile
> > cleanup, I noticed that there are still lots of hacks inside them,
> > that are there from the early days when we adopted Sphinx. Perhaps
> > it could make sense to move part of the logic there to this new
> > build tool, which could, for instance, replace the logic inside
> > scripts/sphinx-pre-install and get rid of some magic at the Makefile
> > like the one which handles SPHINXDIRS.
> >
> > So, at least for now, I would prefer to keep it under scripts.  
> 
> I pretty strongly disagree ... scripts/ is a dumping ground, nobody
> really knows what all that stuff there is, nobody is responsible for it.
> Something under tools/ would be more evident as to its purpose and
> maintainership.  We could maybe just do tools/docs/ and move things like
> sphinx-pre-install there as well...
> 
> Anyway, I won't try to hold up this work based on that, but now you know
> how I feel...:)

A tools/docs with all doc-related tool there is certainly appealing.
Yet, I would move all such scripts on a separate patchset.

Thanks,
Mauro