* [PATCH v4 01/19] scripts/jobserver-exec: move the code to a class
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 02/19] scripts/jobserver-exec: move its class to the lib directory Mauro Carvalho Chehab
` (18 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Convert the code inside jobserver-exec to a class and
properly document it.
Using a class allows reusing the jobserver logic on other
scripts.
While the main code remains unchanged, being compatible with
Python 2.6 and 3.0+, its coding style now follows a more
modern standard, having tabs replaced by a 4-spaces
indent, passing autopep8, black and pylint.
The code now allows allows using a pythonic way to
enter/exit a python code, e.g. it now supports:
with JobserverExec() as jobserver:
jobserver.run(sys.argv[1:])
With the new code, the __exit__() function should ensure
that the jobserver slot will be closed at the end, even if
something bad happens somewhere.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/jobserver-exec | 218 ++++++++++++++++++++++++++++-------------
1 file changed, 151 insertions(+), 67 deletions(-)
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 7eca035472d3..b386b1a845de 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -1,77 +1,161 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
+# pylint: disable=C0103,C0209
+#
# This determines how many parallel tasks "make" is expecting, as it is
# not exposed via an special variables, reserves them all, runs a subprocess
# with PARALLELISM environment variable set, and releases the jobs back again.
#
# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
-from __future__ import print_function
-import os, sys, errno
+
+"""
+Interacts with the POSIX jobserver during the Kernel build time.
+
+A "normal" jobserver task, like the one initiated by a make subrocess would do:
+
+ - open read/write file descriptors to communicate with the job server;
+ - ask for one slot by calling:
+ claim = os.read(reader, 1)
+ - when the job finshes, call:
+ os.write(writer, b"+") # os.write(writer, claim)
+
+Here, the goal is different: This script aims to get the remaining number
+of slots available, using all of them to run a command which handle tasks in
+parallel. To to that, it has a loop that ends only after there are no
+slots left. It then increments the number by one, in order to allow a
+call equivalent to make -j$((claim+1)), e.g. having a parent make creating
+$claim child to do the actual work.
+
+The end goal here is to keep the total number of build tasks under the
+limit established by the initial make -j$n_proc call.
+"""
+
+import errno
+import os
import subprocess
+import sys
-# Extract and prepare jobserver file descriptors from environment.
-claim = 0
-jobs = b""
-try:
- # Fetch the make environment options.
- flags = os.environ['MAKEFLAGS']
-
- # Look for "--jobserver=R,W"
- # Note that GNU Make has used --jobserver-fds and --jobserver-auth
- # so this handles all of them.
- opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
-
- # Parse out R,W file descriptor numbers and set them nonblocking.
- # If the MAKEFLAGS variable contains multiple instances of the
- # --jobserver-auth= option, the last one is relevant.
- fds = opts[-1].split("=", 1)[1]
-
- # Starting with GNU Make 4.4, named pipes are used for reader and writer.
- # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
- _, _, path = fds.partition('fifo:')
-
- if path:
- reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
- writer = os.open(path, os.O_WRONLY)
- else:
- reader, writer = [int(x) for x in fds.split(",", 1)]
- # Open a private copy of reader to avoid setting nonblocking
- # on an unexpecting process with the same reader fd.
- reader = os.open("/proc/self/fd/%d" % (reader),
- os.O_RDONLY | os.O_NONBLOCK)
-
- # Read out as many jobserver slots as possible.
- while True:
- try:
- slot = os.read(reader, 8)
- jobs += slot
- except (OSError, IOError) as e:
- if e.errno == errno.EWOULDBLOCK:
- # Stop at the end of the jobserver queue.
- break
- # If something went wrong, give back the jobs.
- if len(jobs):
- os.write(writer, jobs)
- raise e
- # Add a bump for our caller's reserveration, since we're just going
- # to sit here blocked on our child.
- claim = len(jobs) + 1
-except (KeyError, IndexError, ValueError, OSError, IOError) as e:
- # Any missing environment strings or bad fds should result in just
- # not being parallel.
- pass
-
-# We can only claim parallelism if there was a jobserver (i.e. a top-level
-# "-jN" argument) and there were no other failures. Otherwise leave out the
-# environment variable and let the child figure out what is best.
-if claim > 0:
- os.environ['PARALLELISM'] = '%d' % (claim)
-
-rc = subprocess.call(sys.argv[1:])
-
-# Return all the reserved slots.
-if len(jobs):
- os.write(writer, jobs)
-
-sys.exit(rc)
+
+class JobserverExec:
+ """
+ Claim all slots from make using POSIX Jobserver.
+
+ The main methods here are:
+ - open(): reserves all slots;
+ - close(): method returns all used slots back to make;
+ - run(): executes a command setting PARALLELISM=<available slots jobs + 1>
+ """
+
+ def __init__(self):
+ """Initialize internal vars"""
+ self.claim = 0
+ self.jobs = b""
+ self.reader = None
+ self.writer = None
+ self.is_open = False
+
+ def open(self):
+ """Reserve all available slots to be claimed later on"""
+
+ if self.is_open:
+ return
+
+ try:
+ # Fetch the make environment options.
+ flags = os.environ["MAKEFLAGS"]
+ # Look for "--jobserver=R,W"
+ # Note that GNU Make has used --jobserver-fds and --jobserver-auth
+ # so this handles all of them.
+ opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
+ # Parse out R,W file descriptor numbers and set them nonblocking.
+ # If the MAKEFLAGS variable contains multiple instances of the
+ # --jobserver-auth= option, the last one is relevant.
+ fds = opts[-1].split("=", 1)[1]
+
+ # Starting with GNU Make 4.4, named pipes are used for reader
+ # and writer.
+ # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
+ _, _, path = fds.partition("fifo:")
+
+ if path:
+ self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
+ self.writer = os.open(path, os.O_WRONLY)
+ else:
+ self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
+ # Open a private copy of reader to avoid setting nonblocking
+ # on an unexpecting process with the same reader fd.
+ self.reader = os.open("/proc/self/fd/%d" % (self.reader),
+ os.O_RDONLY | os.O_NONBLOCK)
+
+ # Read out as many jobserver slots as possible
+ while True:
+ try:
+ slot = os.read(self.reader, 8)
+ self.jobs += slot
+ except (OSError, IOError) as e:
+ if e.errno == errno.EWOULDBLOCK:
+ # Stop at the end of the jobserver queue.
+ break
+ # If something went wrong, give back the jobs.
+ if self.jobs:
+ os.write(self.writer, self.jobs)
+ raise e
+
+ # Add a bump for our caller's reserveration, since we're just going
+ # to sit here blocked on our child.
+ self.claim = len(self.jobs) + 1
+
+ except (KeyError, IndexError, ValueError, OSError, IOError):
+ # Any missing environment strings or bad fds should result in just
+ # not being parallel.
+ self.claim = None
+
+ self.is_open = True
+
+ def close(self):
+ """Return all reserved slots to Jobserver"""
+
+ if not self.is_open:
+ return
+
+ # Return all the reserved slots.
+ if len(self.jobs):
+ os.write(self.writer, self.jobs)
+
+ self.is_open = False
+
+ def __enter__(self):
+ self.open()
+ return self
+
+ def __exit__(self, exc_type, exc_value, exc_traceback):
+ self.close()
+
+ def run(self, cmd):
+ """
+ Run a command setting PARALLELISM env variable to the number of
+ available job slots (claim) + 1, e.g. it will reserve claim slots
+ to do the actual build work, plus one to monitor its childs.
+ """
+ self.open() # Ensure that self.claim is set
+
+ # We can only claim parallelism if there was a jobserver (i.e. a
+ # top-level "-jN" argument) and there were no other failures. Otherwise
+ # leave out the environment variable and let the child figure out what
+ # is best.
+ if self.claim:
+ os.environ["PARALLELISM"] = str(self.claim)
+
+ return subprocess.call(cmd)
+
+
+def main():
+ """Main program"""
+ with JobserverExec() as jobserver:
+ jobserver.run(sys.argv[1:])
+
+
+if __name__ == "__main__":
+ main()
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 02/19] scripts/jobserver-exec: move its class to the lib directory
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 01/19] scripts/jobserver-exec: move the code to a class Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 03/19] scripts/jobserver-exec: add a help message Mauro Carvalho Chehab
` (17 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
To make it easier to be re-used, move the JobserverExec class
to the library directory.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/jobserver-exec | 152 +++------------------------------------
scripts/lib/jobserver.py | 149 ++++++++++++++++++++++++++++++++++++++
2 files changed, 160 insertions(+), 141 deletions(-)
create mode 100755 scripts/lib/jobserver.py
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index b386b1a845de..40a0f0058733 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -1,155 +1,25 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
-#
-# pylint: disable=C0103,C0209
-#
-# This determines how many parallel tasks "make" is expecting, as it is
-# not exposed via an special variables, reserves them all, runs a subprocess
-# with PARALLELISM environment variable set, and releases the jobs back again.
-#
-# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
-"""
-Interacts with the POSIX jobserver during the Kernel build time.
-
-A "normal" jobserver task, like the one initiated by a make subrocess would do:
-
- - open read/write file descriptors to communicate with the job server;
- - ask for one slot by calling:
- claim = os.read(reader, 1)
- - when the job finshes, call:
- os.write(writer, b"+") # os.write(writer, claim)
-
-Here, the goal is different: This script aims to get the remaining number
-of slots available, using all of them to run a command which handle tasks in
-parallel. To to that, it has a loop that ends only after there are no
-slots left. It then increments the number by one, in order to allow a
-call equivalent to make -j$((claim+1)), e.g. having a parent make creating
-$claim child to do the actual work.
-
-The end goal here is to keep the total number of build tasks under the
-limit established by the initial make -j$n_proc call.
-"""
-
-import errno
import os
-import subprocess
import sys
+LIB_DIR = "lib"
+SRC_DIR = os.path.dirname(os.path.realpath(__file__))
-class JobserverExec:
- """
- Claim all slots from make using POSIX Jobserver.
+sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
- The main methods here are:
- - open(): reserves all slots;
- - close(): method returns all used slots back to make;
- - run(): executes a command setting PARALLELISM=<available slots jobs + 1>
- """
+from jobserver import JobserverExec # pylint: disable=C0415
- def __init__(self):
- """Initialize internal vars"""
- self.claim = 0
- self.jobs = b""
- self.reader = None
- self.writer = None
- self.is_open = False
- def open(self):
- """Reserve all available slots to be claimed later on"""
-
- if self.is_open:
- return
-
- try:
- # Fetch the make environment options.
- flags = os.environ["MAKEFLAGS"]
- # Look for "--jobserver=R,W"
- # Note that GNU Make has used --jobserver-fds and --jobserver-auth
- # so this handles all of them.
- opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
-
- # Parse out R,W file descriptor numbers and set them nonblocking.
- # If the MAKEFLAGS variable contains multiple instances of the
- # --jobserver-auth= option, the last one is relevant.
- fds = opts[-1].split("=", 1)[1]
-
- # Starting with GNU Make 4.4, named pipes are used for reader
- # and writer.
- # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
- _, _, path = fds.partition("fifo:")
-
- if path:
- self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
- self.writer = os.open(path, os.O_WRONLY)
- else:
- self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
- # Open a private copy of reader to avoid setting nonblocking
- # on an unexpecting process with the same reader fd.
- self.reader = os.open("/proc/self/fd/%d" % (self.reader),
- os.O_RDONLY | os.O_NONBLOCK)
-
- # Read out as many jobserver slots as possible
- while True:
- try:
- slot = os.read(self.reader, 8)
- self.jobs += slot
- except (OSError, IOError) as e:
- if e.errno == errno.EWOULDBLOCK:
- # Stop at the end of the jobserver queue.
- break
- # If something went wrong, give back the jobs.
- if self.jobs:
- os.write(self.writer, self.jobs)
- raise e
-
- # Add a bump for our caller's reserveration, since we're just going
- # to sit here blocked on our child.
- self.claim = len(self.jobs) + 1
-
- except (KeyError, IndexError, ValueError, OSError, IOError):
- # Any missing environment strings or bad fds should result in just
- # not being parallel.
- self.claim = None
-
- self.is_open = True
-
- def close(self):
- """Return all reserved slots to Jobserver"""
-
- if not self.is_open:
- return
-
- # Return all the reserved slots.
- if len(self.jobs):
- os.write(self.writer, self.jobs)
-
- self.is_open = False
-
- def __enter__(self):
- self.open()
- return self
-
- def __exit__(self, exc_type, exc_value, exc_traceback):
- self.close()
-
- def run(self, cmd):
- """
- Run a command setting PARALLELISM env variable to the number of
- available job slots (claim) + 1, e.g. it will reserve claim slots
- to do the actual build work, plus one to monitor its childs.
- """
- self.open() # Ensure that self.claim is set
-
- # We can only claim parallelism if there was a jobserver (i.e. a
- # top-level "-jN" argument) and there were no other failures. Otherwise
- # leave out the environment variable and let the child figure out what
- # is best.
- if self.claim:
- os.environ["PARALLELISM"] = str(self.claim)
-
- return subprocess.call(cmd)
+"""
+Determines how many parallel tasks "make" is expecting, as it is
+not exposed via an special variables, reserves them all, runs a subprocess
+with PARALLELISM environment variable set, and releases the jobs back again.
+See:
+ https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+"""
def main():
"""Main program"""
diff --git a/scripts/lib/jobserver.py b/scripts/lib/jobserver.py
new file mode 100755
index 000000000000..98d8b0ff0c89
--- /dev/null
+++ b/scripts/lib/jobserver.py
@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0+
+#
+# pylint: disable=C0103,C0209
+#
+#
+
+"""
+Interacts with the POSIX jobserver during the Kernel build time.
+
+A "normal" jobserver task, like the one initiated by a make subrocess would do:
+
+ - open read/write file descriptors to communicate with the job server;
+ - ask for one slot by calling:
+ claim = os.read(reader, 1)
+ - when the job finshes, call:
+ os.write(writer, b"+") # os.write(writer, claim)
+
+Here, the goal is different: This script aims to get the remaining number
+of slots available, using all of them to run a command which handle tasks in
+parallel. To to that, it has a loop that ends only after there are no
+slots left. It then increments the number by one, in order to allow a
+call equivalent to make -j$((claim+1)), e.g. having a parent make creating
+$claim child to do the actual work.
+
+The end goal here is to keep the total number of build tasks under the
+limit established by the initial make -j$n_proc call.
+
+See:
+ https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+"""
+
+import errno
+import os
+import subprocess
+import sys
+
+class JobserverExec:
+ """
+ Claim all slots from make using POSIX Jobserver.
+
+ The main methods here are:
+ - open(): reserves all slots;
+ - close(): method returns all used slots back to make;
+ - run(): executes a command setting PARALLELISM=<available slots jobs + 1>
+ """
+
+ def __init__(self):
+ """Initialize internal vars"""
+ self.claim = 0
+ self.jobs = b""
+ self.reader = None
+ self.writer = None
+ self.is_open = False
+
+ def open(self):
+ """Reserve all available slots to be claimed later on"""
+
+ if self.is_open:
+ return
+
+ try:
+ # Fetch the make environment options.
+ flags = os.environ["MAKEFLAGS"]
+ # Look for "--jobserver=R,W"
+ # Note that GNU Make has used --jobserver-fds and --jobserver-auth
+ # so this handles all of them.
+ opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
+ # Parse out R,W file descriptor numbers and set them nonblocking.
+ # If the MAKEFLAGS variable contains multiple instances of the
+ # --jobserver-auth= option, the last one is relevant.
+ fds = opts[-1].split("=", 1)[1]
+
+ # Starting with GNU Make 4.4, named pipes are used for reader
+ # and writer.
+ # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
+ _, _, path = fds.partition("fifo:")
+
+ if path:
+ self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
+ self.writer = os.open(path, os.O_WRONLY)
+ else:
+ self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
+ # Open a private copy of reader to avoid setting nonblocking
+ # on an unexpecting process with the same reader fd.
+ self.reader = os.open("/proc/self/fd/%d" % (self.reader),
+ os.O_RDONLY | os.O_NONBLOCK)
+
+ # Read out as many jobserver slots as possible
+ while True:
+ try:
+ slot = os.read(self.reader, 8)
+ self.jobs += slot
+ except (OSError, IOError) as e:
+ if e.errno == errno.EWOULDBLOCK:
+ # Stop at the end of the jobserver queue.
+ break
+ # If something went wrong, give back the jobs.
+ if self.jobs:
+ os.write(self.writer, self.jobs)
+ raise e
+
+ # Add a bump for our caller's reserveration, since we're just going
+ # to sit here blocked on our child.
+ self.claim = len(self.jobs) + 1
+
+ except (KeyError, IndexError, ValueError, OSError, IOError):
+ # Any missing environment strings or bad fds should result in just
+ # not being parallel.
+ self.claim = None
+
+ self.is_open = True
+
+ def close(self):
+ """Return all reserved slots to Jobserver"""
+
+ if not self.is_open:
+ return
+
+ # Return all the reserved slots.
+ if len(self.jobs):
+ os.write(self.writer, self.jobs)
+
+ self.is_open = False
+
+ def __enter__(self):
+ self.open()
+ return self
+
+ def __exit__(self, exc_type, exc_value, exc_traceback):
+ self.close()
+
+ def run(self, cmd, *args, **pwargs):
+ """
+ Run a command setting PARALLELISM env variable to the number of
+ available job slots (claim) + 1, e.g. it will reserve claim slots
+ to do the actual build work, plus one to monitor its childs.
+ """
+ self.open() # Ensure that self.claim is set
+
+ # We can only claim parallelism if there was a jobserver (i.e. a
+ # top-level "-jN" argument) and there were no other failures. Otherwise
+ # leave out the environment variable and let the child figure out what
+ # is best.
+ if self.claim:
+ os.environ["PARALLELISM"] = str(self.claim)
+
+ return subprocess.call(cmd, *args, **pwargs)
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 03/19] scripts/jobserver-exec: add a help message
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 01/19] scripts/jobserver-exec: move the code to a class Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 02/19] scripts/jobserver-exec: move its class to the lib directory Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs Mauro Carvalho Chehab
` (16 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Currently, calling it without an argument shows an ugly error
message. Instead, print a message using pythondoc as description.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/jobserver-exec | 22 +++++++++++++---------
1 file changed, 13 insertions(+), 9 deletions(-)
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 40a0f0058733..ae23afd344ec 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -1,6 +1,15 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
+"""
+Determines how many parallel tasks "make" is expecting, as it is
+not exposed via any special variables, reserves them all, runs a subprocess
+with PARALLELISM environment variable set, and releases the jobs back again.
+
+See:
+ https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+"""
+
import os
import sys
@@ -12,17 +21,12 @@ sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
from jobserver import JobserverExec # pylint: disable=C0415
-"""
-Determines how many parallel tasks "make" is expecting, as it is
-not exposed via an special variables, reserves them all, runs a subprocess
-with PARALLELISM environment variable set, and releases the jobs back again.
-
-See:
- https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
-"""
-
def main():
"""Main program"""
+ if len(sys.argv) < 2:
+ name = os.path.basename(__file__)
+ sys.exit("usage: " + name +" command [args ...]\n" + __doc__)
+
with JobserverExec() as jobserver:
jobserver.run(sys.argv[1:])
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (2 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 03/19] scripts/jobserver-exec: add a help message Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 16:42 ` Jonathan Corbet
2025-09-04 7:33 ` [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install Mauro Carvalho Chehab
` (15 subsequent siblings)
19 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Akira Yokosawa, Alex Shi, Dongliang Mu,
Federico Vaga, Mauro Carvalho Chehab, Randy Dunlap, Yanteng Si,
linux-kernel
As we're reorganizing the place where doc scripts are located,
move this one to tools/docs.
No functional changes.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 14 +++++++-------
Documentation/doc-guide/sphinx.rst | 4 ++--
Documentation/sphinx/kerneldoc-preamble.sty | 2 +-
.../translations/it_IT/doc-guide/sphinx.rst | 4 ++--
.../translations/zh_CN/doc-guide/sphinx.rst | 4 ++--
Documentation/translations/zh_CN/how-to.rst | 2 +-
MAINTAINERS | 3 +--
{scripts => tools/docs}/sphinx-pre-install | 0
8 files changed, 16 insertions(+), 17 deletions(-)
rename {scripts => tools/docs}/sphinx-pre-install (100%)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 5c20c68be89a..deb2029228ed 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -46,7 +46,7 @@ ifeq ($(HAVE_SPHINX),0)
.DEFAULT:
$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
@echo
- @$(srctree)/scripts/sphinx-pre-install
+ @$(srctree)/tools/docs/sphinx-pre-install
@echo " SKIP Sphinx $@ target."
else # HAVE_SPHINX
@@ -105,7 +105,7 @@ quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
fi
htmldocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
# If Rust support is available and .config exists, add rustdoc generated contents.
@@ -119,7 +119,7 @@ endif
endif
texinfodocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
# Note: the 'info' Make target is generated by sphinx itself when
@@ -131,7 +131,7 @@ linkcheckdocs:
@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
latexdocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
ifeq ($(HAVE_PDFLATEX),0)
@@ -144,7 +144,7 @@ else # HAVE_PDFLATEX
pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
pdfdocs: latexdocs
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
$(foreach var,$(SPHINXDIRS), \
$(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
mkdir -p $(BUILDDIR)/$(var)/pdf; \
@@ -154,11 +154,11 @@ pdfdocs: latexdocs
endif # HAVE_PDFLATEX
epubdocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
xmldocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
endif # HAVE_SPHINX
diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 607589592bfb..932f68c53075 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -106,7 +106,7 @@ There's a script that automatically checks for Sphinx dependencies. If it can
recognize your distribution, it will also give a hint about the install
command line options for your distro::
- $ ./scripts/sphinx-pre-install
+ $ ./tools/docs/sphinx-pre-install
Checking if the needed tools for Fedora release 26 (Twenty Six) are available
Warning: better to also install "texlive-luatex85".
You should run:
@@ -116,7 +116,7 @@ command line options for your distro::
. sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
- Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+ Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-pre-install line 468.
By default, it checks all the requirements for both html and PDF, including
the requirements for images, math expressions and LaTeX build, and assumes
diff --git a/Documentation/sphinx/kerneldoc-preamble.sty b/Documentation/sphinx/kerneldoc-preamble.sty
index 5d68395539fe..16d9ff46fdf6 100644
--- a/Documentation/sphinx/kerneldoc-preamble.sty
+++ b/Documentation/sphinx/kerneldoc-preamble.sty
@@ -220,7 +220,7 @@
If you want them, please install non-variable ``Noto Sans CJK''
font families along with the texlive-xecjk package by following
instructions from
- \sphinxcode{./scripts/sphinx-pre-install}.
+ \sphinxcode{./tools/docs/sphinx-pre-install}.
Having optional non-variable ``Noto Serif CJK'' font families will
improve the looks of those translations.
\end{sphinxadmonition}}
diff --git a/Documentation/translations/it_IT/doc-guide/sphinx.rst b/Documentation/translations/it_IT/doc-guide/sphinx.rst
index 1f513bc33618..a5c5d935febf 100644
--- a/Documentation/translations/it_IT/doc-guide/sphinx.rst
+++ b/Documentation/translations/it_IT/doc-guide/sphinx.rst
@@ -109,7 +109,7 @@ Sphinx. Se lo script riesce a riconoscere la vostra distribuzione, allora
sar�� in grado di darvi dei suggerimenti su come procedere per completare
l'installazione::
- $ ./scripts/sphinx-pre-install
+ $ ./tools/docs/sphinx-pre-install
Checking if the needed tools for Fedora release 26 (Twenty Six) are available
Warning: better to also install "texlive-luatex85".
You should run:
@@ -119,7 +119,7 @@ l'installazione::
. sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
- Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+ Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-pre-install line 468.
L'impostazione predefinita prevede il controllo dei requisiti per la generazione
di documenti html e PDF, includendo anche il supporto per le immagini, le
diff --git a/Documentation/translations/zh_CN/doc-guide/sphinx.rst b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
index 23eac67fbc30..3375c6f3a811 100644
--- a/Documentation/translations/zh_CN/doc-guide/sphinx.rst
+++ b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
@@ -84,7 +84,7 @@ PDF���LaTeX������
������������������������������������Sphinx������������������������������������������������������������������������
������������������::
- $ ./scripts/sphinx-pre-install
+ $ ./tools/docs/sphinx-pre-install
Checking if the needed tools for Fedora release 26 (Twenty Six) are available
Warning: better to also install "texlive-luatex85".
You should run:
@@ -94,7 +94,7 @@ PDF���LaTeX������
. sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
- Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+ Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-pre-install line 468.
������������������������������html���PDF������������������������������������������������������LaTeX���������
���������������������������������Python���������html���������������������������������������������������������
diff --git a/Documentation/translations/zh_CN/how-to.rst b/Documentation/translations/zh_CN/how-to.rst
index ddd99c0f9b4d..714664fec308 100644
--- a/Documentation/translations/zh_CN/how-to.rst
+++ b/Documentation/translations/zh_CN/how-to.rst
@@ -64,7 +64,7 @@ Linux ��������������������������� Linux ���������������������������������������
::
cd linux
- ./scripts/sphinx-pre-install
+ ./tools/docs/sphinx-pre-install
��� Fedora ���������������������������������::
diff --git a/MAINTAINERS b/MAINTAINERS
index ef87548b8f88..06bbed30b788 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7310,7 +7310,6 @@ F: scripts/lib/abi/*
F: scripts/lib/kdoc/*
F: tools/docs/*
F: tools/net/ynl/pyynl/lib/doc_generator.py
-F: scripts/sphinx-pre-install
X: Documentation/ABI/
X: Documentation/admin-guide/media/
X: Documentation/devicetree/
@@ -7345,7 +7344,7 @@ L: linux-doc@vger.kernel.org
S: Maintained
F: Documentation/sphinx/parse-headers.pl
F: scripts/documentation-file-ref-check
-F: scripts/sphinx-pre-install
+F: tools/docs/sphinx-pre-install
DOCUMENTATION/ITALIAN
M: Federico Vaga <federico.vaga@vaga.pv.it>
diff --git a/scripts/sphinx-pre-install b/tools/docs/sphinx-pre-install
similarity index 100%
rename from scripts/sphinx-pre-install
rename to tools/docs/sphinx-pre-install
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* Re: [PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs
2025-09-04 7:33 ` [PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs Mauro Carvalho Chehab
@ 2025-09-04 16:42 ` Jonathan Corbet
2025-09-05 7:39 ` Mauro Carvalho Chehab
2025-09-05 12:25 ` Mauro Carvalho Chehab
0 siblings, 2 replies; 67+ messages in thread
From: Jonathan Corbet @ 2025-09-04 16:42 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Akira Yokosawa, Alex Shi, Dongliang Mu,
Federico Vaga, Mauro Carvalho Chehab, Randy Dunlap, Yanteng Si,
linux-kernel
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> As we're reorganizing the place where doc scripts are located,
> move this one to tools/docs.
>
> No functional changes.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> Documentation/Makefile | 14 +++++++-------
> Documentation/doc-guide/sphinx.rst | 4 ++--
> Documentation/sphinx/kerneldoc-preamble.sty | 2 +-
> .../translations/it_IT/doc-guide/sphinx.rst | 4 ++--
> .../translations/zh_CN/doc-guide/sphinx.rst | 4 ++--
> Documentation/translations/zh_CN/how-to.rst | 2 +-
This one fails to apply to the translations - it looks like some sort of
encoding problem...?
Thanks,
jon
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs
2025-09-04 16:42 ` Jonathan Corbet
@ 2025-09-05 7:39 ` Mauro Carvalho Chehab
2025-09-05 12:25 ` Mauro Carvalho Chehab
1 sibling, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-05 7:39 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Linux Doc Mailing List, Akira Yokosawa, Dongliang Mu,
Federico Vaga, Randy Dunlap, Yanteng Si, linux-kernel
Em Thu, 04 Sep 2025 10:42:56 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
> > As we're reorganizing the place where doc scripts are located,
> > move this one to tools/docs.
> >
> > No functional changes.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > Documentation/Makefile | 14 +++++++-------
> > Documentation/doc-guide/sphinx.rst | 4 ++--
> > Documentation/sphinx/kerneldoc-preamble.sty | 2 +-
> > .../translations/it_IT/doc-guide/sphinx.rst | 4 ++--
> > .../translations/zh_CN/doc-guide/sphinx.rst | 4 ++--
> > Documentation/translations/zh_CN/how-to.rst | 2 +-
>
> This one fails to apply to the translations - it looks like some sort of
> encoding problem...?
Yes. My new mailbomb script in Python mangled it.
I ended sending you a PR, as this way we know for sure that we won't
have any encoding issues on either side.
---
Checking encoding after the patch series:
$ file Documentation/translations/zh_CN/how-to.rst
Documentation/translations/zh_CN/how-to.rst: Unicode text, UTF-8 text
$ python3
Python 3.13.7 (main, Aug 14 2025, 00:00:00) [GCC 15.2.1 20250808 (Red Hat 15.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> with open("Documentation/translations/zh_CN/how-to.rst", "r", encoding="utf-8") as fp: c=fp.read()
...
>>>
And manually looking on its contents, the patch itself is OK.
Looking at the e-mail generated using Python EmailMessage class, however:
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Python completely mangled it by replacing all UTF chars with �:
--- a/Documentation/translations/zh_CN/doc-guide/sphinx.rst
+++ b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
@@ -84,7 +84,7 @@ PDF���LaTeX������
������������������������������������Sphinx������������������������������������������������������������������������
������������������::
- $ ./scripts/sphinx-pre-install
+ $ ./tools/docs/sphinx-pre-install
Checking if the needed tools for Fedora release 26 (Twenty Six) are available
When I have time, I'll try to fix it - or end giving up of the Python
version returning to the Perl one, even with the content set to
utf-8 at set_content() function call.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs
2025-09-04 16:42 ` Jonathan Corbet
2025-09-05 7:39 ` Mauro Carvalho Chehab
@ 2025-09-05 12:25 ` Mauro Carvalho Chehab
1 sibling, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-05 12:25 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Linux Doc Mailing List, Akira Yokosawa, Dongliang Mu,
Federico Vaga, Randy Dunlap, Yanteng Si, linux-kernel
Em Thu, 04 Sep 2025 10:42:56 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
> > As we're reorganizing the place where doc scripts are located,
> > move this one to tools/docs.
> >
> > No functional changes.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > Documentation/Makefile | 14 +++++++-------
> > Documentation/doc-guide/sphinx.rst | 4 ++--
> > Documentation/sphinx/kerneldoc-preamble.sty | 2 +-
> > .../translations/it_IT/doc-guide/sphinx.rst | 4 ++--
> > .../translations/zh_CN/doc-guide/sphinx.rst | 4 ++--
> > Documentation/translations/zh_CN/how-to.rst | 2 +-
>
> This one fails to apply to the translations - it looks like some sort of
> encoding problem...?
Mailbomb script broke it, because Python email parser converts emails
without an explicit charset defined to ascii + invalid utf-8 character,
even when the policy is forcing utf-8.
This is the correct patch:
---
[PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs
As we're reorganizing the place where doc scripts are located,
move this one to tools/docs.
No functional changes.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 14 +++++++-------
Documentation/doc-guide/sphinx.rst | 4 ++--
Documentation/sphinx/kerneldoc-preamble.sty | 2 +-
.../translations/it_IT/doc-guide/sphinx.rst | 4 ++--
.../translations/zh_CN/doc-guide/sphinx.rst | 4 ++--
Documentation/translations/zh_CN/how-to.rst | 2 +-
MAINTAINERS | 3 +--
{scripts => tools/docs}/sphinx-pre-install | 0
8 files changed, 16 insertions(+), 17 deletions(-)
rename {scripts => tools/docs}/sphinx-pre-install (100%)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 5c20c68be89a..deb2029228ed 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -46,7 +46,7 @@ ifeq ($(HAVE_SPHINX),0)
.DEFAULT:
$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
@echo
- @$(srctree)/scripts/sphinx-pre-install
+ @$(srctree)/tools/docs/sphinx-pre-install
@echo " SKIP Sphinx $@ target."
else # HAVE_SPHINX
@@ -105,7 +105,7 @@ quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
fi
htmldocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
# If Rust support is available and .config exists, add rustdoc generated contents.
@@ -119,7 +119,7 @@ endif
endif
texinfodocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
# Note: the 'info' Make target is generated by sphinx itself when
@@ -131,7 +131,7 @@ linkcheckdocs:
@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
latexdocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
ifeq ($(HAVE_PDFLATEX),0)
@@ -144,7 +144,7 @@ else # HAVE_PDFLATEX
pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
pdfdocs: latexdocs
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
$(foreach var,$(SPHINXDIRS), $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; mkdir -p $(BUILDDIR)/$(var)/pdf; @@ -154,11 +154,11 @@ pdfdocs: latexdocs
endif # HAVE_PDFLATEX
epubdocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
xmldocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
+ @$(srctree)/tools/docs/sphinx-pre-install --version-check
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
endif # HAVE_SPHINX
diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 607589592bfb..932f68c53075 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -106,7 +106,7 @@ There's a script that automatically checks for Sphinx dependencies. If it can
recognize your distribution, it will also give a hint about the install
command line options for your distro::
- $ ./scripts/sphinx-pre-install
+ $ ./tools/docs/sphinx-pre-install
Checking if the needed tools for Fedora release 26 (Twenty Six) are available
Warning: better to also install "texlive-luatex85".
You should run:
@@ -116,7 +116,7 @@ command line options for your distro::
. sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
- Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+ Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-pre-install line 468.
By default, it checks all the requirements for both html and PDF, including
the requirements for images, math expressions and LaTeX build, and assumes
diff --git a/Documentation/sphinx/kerneldoc-preamble.sty b/Documentation/sphinx/kerneldoc-preamble.sty
index 5d68395539fe..16d9ff46fdf6 100644
--- a/Documentation/sphinx/kerneldoc-preamble.sty
+++ b/Documentation/sphinx/kerneldoc-preamble.sty
@@ -220,7 +220,7 @@
If you want them, please install non-variable ``Noto Sans CJK''
font families along with the texlive-xecjk package by following
instructions from
- \sphinxcode{./scripts/sphinx-pre-install}.
+ \sphinxcode{./tools/docs/sphinx-pre-install}.
Having optional non-variable ``Noto Serif CJK'' font families will
improve the looks of those translations.
\end{sphinxadmonition}}
diff --git a/Documentation/translations/it_IT/doc-guide/sphinx.rst b/Documentation/translations/it_IT/doc-guide/sphinx.rst
index 1f513bc33618..a5c5d935febf 100644
--- a/Documentation/translations/it_IT/doc-guide/sphinx.rst
+++ b/Documentation/translations/it_IT/doc-guide/sphinx.rst
@@ -109,7 +109,7 @@ Sphinx. Se lo script riesce a riconoscere la vostra distribuzione, allora
sar� in grado di darvi dei suggerimenti su come procedere per completare
l'installazione::
- $ ./scripts/sphinx-pre-install
+ $ ./tools/docs/sphinx-pre-install
Checking if the needed tools for Fedora release 26 (Twenty Six) are available
Warning: better to also install "texlive-luatex85".
You should run:
@@ -119,7 +119,7 @@ l'installazione::
. sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
- Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+ Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-pre-install line 468.
L'impostazione predefinita prevede il controllo dei requisiti per la generazione
di documenti html e PDF, includendo anche il supporto per le immagini, le
diff --git a/Documentation/translations/zh_CN/doc-guide/sphinx.rst b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
index 23eac67fbc30..3375c6f3a811 100644
--- a/Documentation/translations/zh_CN/doc-guide/sphinx.rst
+++ b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
@@ -84,7 +84,7 @@ PDF和LaTeX构建
这有一个脚本可以自动检查Sphinx依赖项。如果它认得您的发行版，还会提示您所用发行
版的安装命令::
- $ ./scripts/sphinx-pre-install
+ $ ./tools/docs/sphinx-pre-install
Checking if the needed tools for Fedora release 26 (Twenty Six) are available
Warning: better to also install "texlive-luatex85".
You should run:
@@ -94,7 +94,7 @@ PDF和LaTeX构建
. sphinx_2.4.4/bin/activate
pip install -r Documentation/sphinx/requirements.txt
- Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+ Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-pre-install line 468.
默认情况下，它会检查html和PDF的所有依赖项，包括图像、数学表达式和LaTeX构建的
需求，并假设将使用虚拟Python环境。html构建所需的依赖项被认为是必需的，其他依
diff --git a/Documentation/translations/zh_CN/how-to.rst b/Documentation/translations/zh_CN/how-to.rst
index ddd99c0f9b4d..714664fec308 100644
--- a/Documentation/translations/zh_CN/how-to.rst
+++ b/Documentation/translations/zh_CN/how-to.rst
@@ -64,7 +64,7 @@ Linux 发行版和简单地使用 Linux 命令行，那么可以迅速开始了
::
cd linux
- ./scripts/sphinx-pre-install
+ ./tools/docs/sphinx-pre-install
以 Fedora 为例，它的输出是这样的::
diff --git a/MAINTAINERS b/MAINTAINERS
index ef87548b8f88..06bbed30b788 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7310,7 +7310,6 @@ F: scripts/lib/abi/*
F: scripts/lib/kdoc/*
F: tools/docs/*
F: tools/net/ynl/pyynl/lib/doc_generator.py
-F: scripts/sphinx-pre-install
X: Documentation/ABI/
X: Documentation/admin-guide/media/
X: Documentation/devicetree/
@@ -7345,7 +7344,7 @@ L: linux-doc@vger.kernel.org
S: Maintained
F: Documentation/sphinx/parse-headers.pl
F: scripts/documentation-file-ref-check
-F: scripts/sphinx-pre-install
+F: tools/docs/sphinx-pre-install
DOCUMENTATION/ITALIAN
M: Federico Vaga <federico.vaga@vaga.pv.it>
diff --git a/scripts/sphinx-pre-install b/tools/docs/sphinx-pre-install
similarity index 100%
rename from scripts/sphinx-pre-install
rename to tools/docs/sphinx-pre-install
--
2.51.0
Thanks,
Mauro
^ permalink raw reply related [flat|nested] 67+ messages in thread
* [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (3 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 04/19] scripts: sphinx-pre-install: move it to tools/docs Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-10 10:14 ` Jani Nikula
2025-09-04 7:33 ` [PATCH v4 06/19] tools/docs: python_version: drop a debug print Mauro Carvalho Chehab
` (14 subsequent siblings)
19 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
The sphinx-pre-install code has some logic to deal with Python
version, which ensures that a minimal version will be enforced
for documentation build logic.
Move it to a separate library to allow re-using its code.
No functional changes.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/lib/python_version.py | 133 +++++++++++++++++++++++++++++++
tools/docs/sphinx-pre-install | 120 +++-------------------------
2 files changed, 146 insertions(+), 107 deletions(-)
create mode 100644 tools/docs/lib/python_version.py
diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_version.py
new file mode 100644
index 000000000000..0519d524e547
--- /dev/null
+++ b/tools/docs/lib/python_version.py
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2017-2025 Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
+
+"""
+Handle Python version check logic.
+
+Not all Python versions are supported by scripts. Yet, on some cases,
+like during documentation build, a newer version of python could be
+available.
+
+This class allows checking if the minimal requirements are followed.
+
+Better than that, PythonVersion.check_python() not only checks the minimal
+requirements, but it automatically switches to a the newest available
+Python version if present.
+
+"""
+
+import os
+import re
+import subprocess
+import sys
+
+from glob import glob
+
+class PythonVersion:
+ """
+ Ancillary methods that checks for missing dependencies for different
+ types of types, like binaries, python modules, rpm deps, etc.
+ """
+
+ def __init__(self, version):
+ """��nitialize self.version tuple from a version string"""
+ self.version = self.parse_version(version)
+
+ @staticmethod
+ def parse_version(version):
+ """Convert a major.minor.patch version into a tuple"""
+ return tuple(int(x) for x in version.split("."))
+
+ @staticmethod
+ def ver_str(version):
+ """Returns a version tuple as major.minor.patch"""
+ return ".".join([str(x) for x in version])
+
+ def __str__(self):
+ """Returns a version tuple as major.minor.patch from self.version"""
+ return self.ver_str(self.version)
+
+ @staticmethod
+ def get_python_version(cmd):
+ """
+ Get python version from a Python binary. As we need to detect if
+ are out there newer python binaries, we can't rely on sys.release here.
+ """
+
+ kwargs = {}
+ if sys.version_info < (3, 7):
+ kwargs['universal_newlines'] = True
+ else:
+ kwargs['text'] = True
+
+ result = subprocess.run([cmd, "--version"],
+ stdout = subprocess.PIPE,
+ stderr = subprocess.PIPE,
+ **kwargs, check=False)
+
+ version = result.stdout.strip()
+
+ match = re.search(r"(\d+\.\d+\.\d+)", version)
+ if match:
+ return PythonVersion.parse_version(match.group(1))
+
+ print(f"Can't parse version {version}")
+ return (0, 0, 0)
+
+ @staticmethod
+ def find_python(min_version):
+ """
+ Detect if are out there any python 3.xy version newer than the
+ current one.
+
+ Note: this routine is limited to up to 2 digits for python3. We
+ may need to update it one day, hopefully on a distant future.
+ """
+ patterns = [
+ "python3.[0-9]",
+ "python3.[0-9][0-9]",
+ ]
+
+ # Seek for a python binary newer than min_version
+ for path in os.getenv("PATH", "").split(":"):
+ for pattern in patterns:
+ for cmd in glob(os.path.join(path, pattern)):
+ if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
+ version = PythonVersion.get_python_version(cmd)
+ if version >= min_version:
+ return cmd
+
+ return None
+
+ @staticmethod
+ def check_python(min_version):
+ """
+ Check if the current python binary satisfies our minimal requirement
+ for Sphinx build. If not, re-run with a newer version if found.
+ """
+ cur_ver = sys.version_info[:3]
+ if cur_ver >= min_version:
+ ver = PythonVersion.ver_str(cur_ver)
+ print(f"Python version: {ver}")
+
+ return
+
+ python_ver = PythonVersion.ver_str(cur_ver)
+
+ new_python_cmd = PythonVersion.find_python(min_version)
+ if not new_python_cmd:
+ print(f"ERROR: Python version {python_ver} is not spported anymore\n")
+ print(" Can't find a new version. This script may fail")
+ return
+
+ # Restart script using the newer version
+ script_path = os.path.abspath(sys.argv[0])
+ args = [new_python_cmd, script_path] + sys.argv[1:]
+
+ print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
+
+ try:
+ os.execv(new_python_cmd, args)
+ except OSError as e:
+ sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install
index 954ed3dc0645..d6d673b7945c 100755
--- a/tools/docs/sphinx-pre-install
+++ b/tools/docs/sphinx-pre-install
@@ -32,20 +32,10 @@ import subprocess
import sys
from glob import glob
+from lib.python_version import PythonVersion
-def parse_version(version):
- """Convert a major.minor.patch version into a tuple"""
- return tuple(int(x) for x in version.split("."))
-
-
-def ver_str(version):
- """Returns a version tuple as major.minor.patch"""
-
- return ".".join([str(x) for x in version])
-
-
-RECOMMENDED_VERSION = parse_version("3.4.3")
-MIN_PYTHON_VERSION = parse_version("3.7")
+RECOMMENDED_VERSION = PythonVersion("3.4.3").version
+MIN_PYTHON_VERSION = PythonVersion("3.7").version
class DepManager:
@@ -235,95 +225,11 @@ class AncillaryMethods:
return None
- @staticmethod
- def get_python_version(cmd):
- """
- Get python version from a Python binary. As we need to detect if
- are out there newer python binaries, we can't rely on sys.release here.
- """
-
- result = SphinxDependencyChecker.run([cmd, "--version"],
- capture_output=True, text=True)
- version = result.stdout.strip()
-
- match = re.search(r"(\d+\.\d+\.\d+)", version)
- if match:
- return parse_version(match.group(1))
-
- print(f"Can't parse version {version}")
- return (0, 0, 0)
-
- @staticmethod
- def find_python():
- """
- Detect if are out there any python 3.xy version newer than the
- current one.
-
- Note: this routine is limited to up to 2 digits for python3. We
- may need to update it one day, hopefully on a distant future.
- """
- patterns = [
- "python3.[0-9]",
- "python3.[0-9][0-9]",
- ]
-
- # Seek for a python binary newer than MIN_PYTHON_VERSION
- for path in os.getenv("PATH", "").split(":"):
- for pattern in patterns:
- for cmd in glob(os.path.join(path, pattern)):
- if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
- version = SphinxDependencyChecker.get_python_version(cmd)
- if version >= MIN_PYTHON_VERSION:
- return cmd
-
- @staticmethod
- def check_python():
- """
- Check if the current python binary satisfies our minimal requirement
- for Sphinx build. If not, re-run with a newer version if found.
- """
- cur_ver = sys.version_info[:3]
- if cur_ver >= MIN_PYTHON_VERSION:
- ver = ver_str(cur_ver)
- print(f"Python version: {ver}")
-
- # This could be useful for debugging purposes
- if SphinxDependencyChecker.which("docutils"):
- result = SphinxDependencyChecker.run(["docutils", "--version"],
- capture_output=True, text=True)
- ver = result.stdout.strip()
- match = re.search(r"(\d+\.\d+\.\d+)", ver)
- if match:
- ver = match.group(1)
-
- print(f"Docutils version: {ver}")
-
- return
-
- python_ver = ver_str(cur_ver)
-
- new_python_cmd = SphinxDependencyChecker.find_python()
- if not new_python_cmd:
- print(f"ERROR: Python version {python_ver} is not spported anymore\n")
- print(" Can't find a new version. This script may fail")
- return
-
- # Restart script using the newer version
- script_path = os.path.abspath(sys.argv[0])
- args = [new_python_cmd, script_path] + sys.argv[1:]
-
- print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
-
- try:
- os.execv(new_python_cmd, args)
- except OSError as e:
- sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
-
@staticmethod
def run(*args, **kwargs):
"""
Excecute a command, hiding its output by default.
- Preserve comatibility with older Python versions.
+ Preserve compatibility with older Python versions.
"""
capture_output = kwargs.pop('capture_output', False)
@@ -527,11 +433,11 @@ class MissingCheckers(AncillaryMethods):
for line in result.stdout.split("\n"):
match = re.match(r"^sphinx-build\s+([\d\.]+)(?:\+(?:/[\da-f]+)|b\d+)?\s*$", line)
if match:
- return parse_version(match.group(1))
+ return PythonVersion.parse_version(match.group(1))
match = re.match(r"^Sphinx.*\s+([\d\.]+)\s*$", line)
if match:
- return parse_version(match.group(1))
+ return PythonVersion.parse_version(match.group(1))
def check_sphinx(self, conf):
"""
@@ -542,7 +448,7 @@ class MissingCheckers(AncillaryMethods):
for line in f:
match = re.match(r"^\s*needs_sphinx\s*=\s*[\'\"]([\d\.]+)[\'\"]", line)
if match:
- self.min_version = parse_version(match.group(1))
+ self.min_version = PythonVersion.parse_version(match.group(1))
break
except IOError:
sys.exit(f"Can't open {conf}")
@@ -562,8 +468,8 @@ class MissingCheckers(AncillaryMethods):
sys.exit(f"{sphinx} didn't return its version")
if self.cur_version < self.min_version:
- curver = ver_str(self.cur_version)
- minver = ver_str(self.min_version)
+ curver = PythonVersion.ver_str(self.cur_version)
+ minver = PythonVersion.ver_str(self.min_version)
print(f"ERROR: Sphinx version is {curver}. It should be >= {minver}")
self.need_sphinx = 1
@@ -1304,7 +1210,7 @@ class SphinxDependencyChecker(MissingCheckers):
else:
if self.need_sphinx and ver >= self.min_version:
return (f, ver)
- elif parse_version(ver) > self.cur_version:
+ elif PythonVersion.parse_version(ver) > self.cur_version:
return (f, ver)
return ("", ver)
@@ -1411,7 +1317,7 @@ class SphinxDependencyChecker(MissingCheckers):
return
if self.latest_avail_ver:
- latest_avail_ver = ver_str(self.latest_avail_ver)
+ latest_avail_ver = PythonVersion.ver_str(self.latest_avail_ver)
if not self.need_sphinx:
# sphinx-build is present and its version is >= $min_version
@@ -1507,7 +1413,7 @@ class SphinxDependencyChecker(MissingCheckers):
else:
print("Unknown OS")
if self.cur_version != (0, 0, 0):
- ver = ver_str(self.cur_version)
+ ver = PythonVersion.ver_str(self.cur_version)
print(f"Sphinx version: {ver}\n")
# Check the type of virtual env, depending on Python version
@@ -1613,7 +1519,7 @@ def main():
checker = SphinxDependencyChecker(args)
- checker.check_python()
+ PythonVersion.check_python(MIN_PYTHON_VERSION)
checker.check_needs()
# Call main if not used as module
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* Re: [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install
2025-09-04 7:33 ` [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install Mauro Carvalho Chehab
@ 2025-09-10 10:14 ` Jani Nikula
2025-09-10 12:24 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-10 10:14 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> The sphinx-pre-install code has some logic to deal with Python
> version, which ensures that a minimal version will be enforced
> for documentation build logic.
>
> Move it to a separate library to allow re-using its code.
>
> No functional changes.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> tools/docs/lib/python_version.py | 133 +++++++++++++++++++++++++++++++
> tools/docs/sphinx-pre-install | 120 +++-------------------------
> 2 files changed, 146 insertions(+), 107 deletions(-)
> create mode 100644 tools/docs/lib/python_version.py
>
> diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_version.py
> new file mode 100644
> index 000000000000..0519d524e547
> --- /dev/null
> +++ b/tools/docs/lib/python_version.py
> @@ -0,0 +1,133 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (c) 2017-2025 Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> +
> +"""
> +Handle Python version check logic.
> +
> +Not all Python versions are supported by scripts. Yet, on some cases,
> +like during documentation build, a newer version of python could be
> +available.
> +
> +This class allows checking if the minimal requirements are followed.
> +
> +Better than that, PythonVersion.check_python() not only checks the minimal
> +requirements, but it automatically switches to a the newest available
> +Python version if present.
> +
> +"""
> +
> +import os
> +import re
> +import subprocess
> +import sys
> +
> +from glob import glob
> +
> +class PythonVersion:
> + """
> + Ancillary methods that checks for missing dependencies for different
> + types of types, like binaries, python modules, rpm deps, etc.
> + """
> +
> + def __init__(self, version):
> + """��nitialize self.version tuple from a version string"""
> + self.version = self.parse_version(version)
> +
> + @staticmethod
> + def parse_version(version):
> + """Convert a major.minor.patch version into a tuple"""
> + return tuple(int(x) for x in version.split("."))
I've written a few quick and dirty semver parsers myself, and it saddens
me to think of adding a simplistic one in the kernel.
I'm just wondering, are we doomed to reinventing the wheels in our
reluctance to depend on anything else?
> +
> + @staticmethod
> + def ver_str(version):
> + """Returns a version tuple as major.minor.patch"""
> + return ".".join([str(x) for x in version])
> +
> + def __str__(self):
> + """Returns a version tuple as major.minor.patch from self.version"""
> + return self.ver_str(self.version)
> +
> + @staticmethod
> + def get_python_version(cmd):
> + """
> + Get python version from a Python binary. As we need to detect if
> + are out there newer python binaries, we can't rely on sys.release here.
> + """
> +
> + kwargs = {}
> + if sys.version_info < (3, 7):
Checking for things that EOL'd four years ago. Why are we doing this to
ourselves? Why should we take on maintenance of code that jumps through
hoops for things that nobody supports anymore?
In Documentation/process/changes.rst we've declared Python 3.9 as
minimum, which is also the oldest version supported by upstream (until
next month). Even Debian oldoldstable (that's two olds) has 3.9.
We're talking about the documentation build. I can undertand being more
conservative about the kernel build in general, but IMHO this is just
extra work for absolutely no users out there. And I'm not advocating for
bleeding edge here.
We could just throw out a lot of crap by setting newer but still
moderately concervative required Python (and Sphinx) versions, and bail
out on older version. Let the user figure out how to get them.
We don't do this for any other tools either.
I'm saying that instead of refactoring this overgrown logic to a
separate file and class, it should be nuked out of the kernel
completely.
> + kwargs['universal_newlines'] = True
> + else:
> + kwargs['text'] = True
> +
> + result = subprocess.run([cmd, "--version"],
> + stdout = subprocess.PIPE,
> + stderr = subprocess.PIPE,
> + **kwargs, check=False)
> +
> + version = result.stdout.strip()
> +
> + match = re.search(r"(\d+\.\d+\.\d+)", version)
> + if match:
> + return PythonVersion.parse_version(match.group(1))
> +
> + print(f"Can't parse version {version}")
> + return (0, 0, 0)
> +
> + @staticmethod
> + def find_python(min_version):
> + """
> + Detect if are out there any python 3.xy version newer than the
> + current one.
> +
> + Note: this routine is limited to up to 2 digits for python3. We
> + may need to update it one day, hopefully on a distant future.
> + """
> + patterns = [
> + "python3.[0-9]",
> + "python3.[0-9][0-9]",
> + ]
> +
> + # Seek for a python binary newer than min_version
> + for path in os.getenv("PATH", "").split(":"):
> + for pattern in patterns:
> + for cmd in glob(os.path.join(path, pattern)):
> + if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
> + version = PythonVersion.get_python_version(cmd)
> + if version >= min_version:
> + return cmd
> +
> + return None
> +
> + @staticmethod
> + def check_python(min_version):
> + """
> + Check if the current python binary satisfies our minimal requirement
> + for Sphinx build. If not, re-run with a newer version if found.
> + """
> + cur_ver = sys.version_info[:3]
> + if cur_ver >= min_version:
> + ver = PythonVersion.ver_str(cur_ver)
> + print(f"Python version: {ver}")
> +
> + return
> +
> + python_ver = PythonVersion.ver_str(cur_ver)
> +
> + new_python_cmd = PythonVersion.find_python(min_version)
> + if not new_python_cmd:
> + print(f"ERROR: Python version {python_ver} is not spported anymore\n")
> + print(" Can't find a new version. This script may fail")
> + return
> +
> + # Restart script using the newer version
I thought the whole idea of restarting was completely rejected by
approximately everyone?!
BR,
Jani.
> + script_path = os.path.abspath(sys.argv[0])
> + args = [new_python_cmd, script_path] + sys.argv[1:]
> +
> + print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
> +
> + try:
> + os.execv(new_python_cmd, args)
> + except OSError as e:
> + sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
> diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install
> index 954ed3dc0645..d6d673b7945c 100755
> --- a/tools/docs/sphinx-pre-install
> +++ b/tools/docs/sphinx-pre-install
> @@ -32,20 +32,10 @@ import subprocess
> import sys
> from glob import glob
>
> +from lib.python_version import PythonVersion
>
> -def parse_version(version):
> - """Convert a major.minor.patch version into a tuple"""
> - return tuple(int(x) for x in version.split("."))
> -
> -
> -def ver_str(version):
> - """Returns a version tuple as major.minor.patch"""
> -
> - return ".".join([str(x) for x in version])
> -
> -
> -RECOMMENDED_VERSION = parse_version("3.4.3")
> -MIN_PYTHON_VERSION = parse_version("3.7")
> +RECOMMENDED_VERSION = PythonVersion("3.4.3").version
> +MIN_PYTHON_VERSION = PythonVersion("3.7").version
>
>
> class DepManager:
> @@ -235,95 +225,11 @@ class AncillaryMethods:
>
> return None
>
> - @staticmethod
> - def get_python_version(cmd):
> - """
> - Get python version from a Python binary. As we need to detect if
> - are out there newer python binaries, we can't rely on sys.release here.
> - """
> -
> - result = SphinxDependencyChecker.run([cmd, "--version"],
> - capture_output=True, text=True)
> - version = result.stdout.strip()
> -
> - match = re.search(r"(\d+\.\d+\.\d+)", version)
> - if match:
> - return parse_version(match.group(1))
> -
> - print(f"Can't parse version {version}")
> - return (0, 0, 0)
> -
> - @staticmethod
> - def find_python():
> - """
> - Detect if are out there any python 3.xy version newer than the
> - current one.
> -
> - Note: this routine is limited to up to 2 digits for python3. We
> - may need to update it one day, hopefully on a distant future.
> - """
> - patterns = [
> - "python3.[0-9]",
> - "python3.[0-9][0-9]",
> - ]
> -
> - # Seek for a python binary newer than MIN_PYTHON_VERSION
> - for path in os.getenv("PATH", "").split(":"):
> - for pattern in patterns:
> - for cmd in glob(os.path.join(path, pattern)):
> - if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
> - version = SphinxDependencyChecker.get_python_version(cmd)
> - if version >= MIN_PYTHON_VERSION:
> - return cmd
> -
> - @staticmethod
> - def check_python():
> - """
> - Check if the current python binary satisfies our minimal requirement
> - for Sphinx build. If not, re-run with a newer version if found.
> - """
> - cur_ver = sys.version_info[:3]
> - if cur_ver >= MIN_PYTHON_VERSION:
> - ver = ver_str(cur_ver)
> - print(f"Python version: {ver}")
> -
> - # This could be useful for debugging purposes
> - if SphinxDependencyChecker.which("docutils"):
> - result = SphinxDependencyChecker.run(["docutils", "--version"],
> - capture_output=True, text=True)
> - ver = result.stdout.strip()
> - match = re.search(r"(\d+\.\d+\.\d+)", ver)
> - if match:
> - ver = match.group(1)
> -
> - print(f"Docutils version: {ver}")
> -
> - return
> -
> - python_ver = ver_str(cur_ver)
> -
> - new_python_cmd = SphinxDependencyChecker.find_python()
> - if not new_python_cmd:
> - print(f"ERROR: Python version {python_ver} is not spported anymore\n")
> - print(" Can't find a new version. This script may fail")
> - return
> -
> - # Restart script using the newer version
> - script_path = os.path.abspath(sys.argv[0])
> - args = [new_python_cmd, script_path] + sys.argv[1:]
> -
> - print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
> -
> - try:
> - os.execv(new_python_cmd, args)
> - except OSError as e:
> - sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
> -
> @staticmethod
> def run(*args, **kwargs):
> """
> Excecute a command, hiding its output by default.
> - Preserve comatibility with older Python versions.
> + Preserve compatibility with older Python versions.
> """
>
> capture_output = kwargs.pop('capture_output', False)
> @@ -527,11 +433,11 @@ class MissingCheckers(AncillaryMethods):
> for line in result.stdout.split("\n"):
> match = re.match(r"^sphinx-build\s+([\d\.]+)(?:\+(?:/[\da-f]+)|b\d+)?\s*$", line)
> if match:
> - return parse_version(match.group(1))
> + return PythonVersion.parse_version(match.group(1))
>
> match = re.match(r"^Sphinx.*\s+([\d\.]+)\s*$", line)
> if match:
> - return parse_version(match.group(1))
> + return PythonVersion.parse_version(match.group(1))
>
> def check_sphinx(self, conf):
> """
> @@ -542,7 +448,7 @@ class MissingCheckers(AncillaryMethods):
> for line in f:
> match = re.match(r"^\s*needs_sphinx\s*=\s*[\'\"]([\d\.]+)[\'\"]", line)
> if match:
> - self.min_version = parse_version(match.group(1))
> + self.min_version = PythonVersion.parse_version(match.group(1))
> break
> except IOError:
> sys.exit(f"Can't open {conf}")
> @@ -562,8 +468,8 @@ class MissingCheckers(AncillaryMethods):
> sys.exit(f"{sphinx} didn't return its version")
>
> if self.cur_version < self.min_version:
> - curver = ver_str(self.cur_version)
> - minver = ver_str(self.min_version)
> + curver = PythonVersion.ver_str(self.cur_version)
> + minver = PythonVersion.ver_str(self.min_version)
>
> print(f"ERROR: Sphinx version is {curver}. It should be >= {minver}")
> self.need_sphinx = 1
> @@ -1304,7 +1210,7 @@ class SphinxDependencyChecker(MissingCheckers):
> else:
> if self.need_sphinx and ver >= self.min_version:
> return (f, ver)
> - elif parse_version(ver) > self.cur_version:
> + elif PythonVersion.parse_version(ver) > self.cur_version:
> return (f, ver)
>
> return ("", ver)
> @@ -1411,7 +1317,7 @@ class SphinxDependencyChecker(MissingCheckers):
> return
>
> if self.latest_avail_ver:
> - latest_avail_ver = ver_str(self.latest_avail_ver)
> + latest_avail_ver = PythonVersion.ver_str(self.latest_avail_ver)
>
> if not self.need_sphinx:
> # sphinx-build is present and its version is >= $min_version
> @@ -1507,7 +1413,7 @@ class SphinxDependencyChecker(MissingCheckers):
> else:
> print("Unknown OS")
> if self.cur_version != (0, 0, 0):
> - ver = ver_str(self.cur_version)
> + ver = PythonVersion.ver_str(self.cur_version)
> print(f"Sphinx version: {ver}\n")
>
> # Check the type of virtual env, depending on Python version
> @@ -1613,7 +1519,7 @@ def main():
>
> checker = SphinxDependencyChecker(args)
>
> - checker.check_python()
> + PythonVersion.check_python(MIN_PYTHON_VERSION)
> checker.check_needs()
>
> # Call main if not used as module
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install
2025-09-10 10:14 ` Jani Nikula
@ 2025-09-10 12:24 ` Mauro Carvalho Chehab
2025-09-11 10:28 ` Jani Nikula
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-10 12:24 UTC (permalink / raw)
To: Jani Nikula; +Cc: Jonathan Corbet, Linux Doc Mailing List, linux-kernel
Em Wed, 10 Sep 2025 13:14:33 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > The sphinx-pre-install code has some logic to deal with Python
> > version, which ensures that a minimal version will be enforced
> > for documentation build logic.
> >
> > Move it to a separate library to allow re-using its code.
> >
> > No functional changes.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > tools/docs/lib/python_version.py | 133 +++++++++++++++++++++++++++++++
> > tools/docs/sphinx-pre-install | 120 +++-------------------------
> > 2 files changed, 146 insertions(+), 107 deletions(-)
> > create mode 100644 tools/docs/lib/python_version.py
> >
> > diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_version.py
> > new file mode 100644
> > index 000000000000..0519d524e547
> > --- /dev/null
> > +++ b/tools/docs/lib/python_version.py
> > @@ -0,0 +1,133 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: GPL-2.0-or-later
> > +# Copyright (c) 2017-2025 Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > +
> > +"""
> > +Handle Python version check logic.
> > +
> > +Not all Python versions are supported by scripts. Yet, on some cases,
> > +like during documentation build, a newer version of python could be
> > +available.
> > +
> > +This class allows checking if the minimal requirements are followed.
> > +
> > +Better than that, PythonVersion.check_python() not only checks the minimal
> > +requirements, but it automatically switches to a the newest available
> > +Python version if present.
> > +
> > +"""
> > +
> > +import os
> > +import re
> > +import subprocess
> > +import sys
> > +
> > +from glob import glob
> > +
> > +class PythonVersion:
> > + """
> > + Ancillary methods that checks for missing dependencies for different
> > + types of types, like binaries, python modules, rpm deps, etc.
> > + """
> > +
> > + def __init__(self, version):
> > + """��nitialize self.version tuple from a version string"""
> > + self.version = self.parse_version(version)
> > +
> > + @staticmethod
> > + def parse_version(version):
> > + """Convert a major.minor.patch version into a tuple"""
> > + return tuple(int(x) for x in version.split("."))
>
> I've written a few quick and dirty semver parsers myself, and it saddens
> me to think of adding a simplistic one in the kernel.
>
> I'm just wondering, are we doomed to reinventing the wheels in our
> reluctance to depend on anything else?
What do you propose instead, using only internal libs(*)?
In any case, import a library just for one or two single-line
functions seem overkill to me.
(*) As this is used by sphinx-pre-install, which is the script which
checks for missing dependencies, whatever we pick, it should not
use external libs.
>
> > +
> > + @staticmethod
> > + def ver_str(version):
> > + """Returns a version tuple as major.minor.patch"""
> > + return ".".join([str(x) for x in version])
> > +
> > + def __str__(self):
> > + """Returns a version tuple as major.minor.patch from self.version"""
> > + return self.ver_str(self.version)
> > +
> > + @staticmethod
> > + def get_python_version(cmd):
> > + """
> > + Get python version from a Python binary. As we need to detect if
> > + are out there newer python binaries, we can't rely on sys.release here.
> > + """
> > +
> > + kwargs = {}
> > + if sys.version_info < (3, 7):
>
> Checking for things that EOL'd four years ago. Why are we doing this to
> ourselves? Why should we take on maintenance of code that jumps through
> hoops for things that nobody supports anymore?
>
> In Documentation/process/changes.rst we've declared Python 3.9 as
> minimum, which is also the oldest version supported by upstream (until
> next month). Even Debian oldoldstable (that's two olds) has 3.9.
>
> We're talking about the documentation build. I can undertand being more
> conservative about the kernel build in general, but IMHO this is just
> extra work for absolutely no users out there. And I'm not advocating for
> bleeding edge here.
>
> We could just throw out a lot of crap by setting newer but still
> moderately concervative required Python (and Sphinx) versions, and bail
> out on older version. Let the user figure out how to get them.
>
> We don't do this for any other tools either.
>
> I'm saying that instead of refactoring this overgrown logic to a
> separate file and class, it should be nuked out of the kernel
> completely.
True, but latest SUSE and openSUSE distros (not counting Thumbleweed
rolling version one) still have Python 3.6 as the main version.
They provide 3.9 as well, but the detection script needs to work with
3.6 to discover that.
If we don't have something like that, we should probably return
using the Perl version of sphinx-pre-install script, which is
a lot more compatible with different distros.
> > + kwargs['universal_newlines'] = True
> > + else:
> > + kwargs['text'] = True
> > +
> > + result = subprocess.run([cmd, "--version"],
> > + stdout = subprocess.PIPE,
> > + stderr = subprocess.PIPE,
> > + **kwargs, check=False)
> > +
> > + version = result.stdout.strip()
> > +
> > + match = re.search(r"(\d+\.\d+\.\d+)", version)
> > + if match:
> > + return PythonVersion.parse_version(match.group(1))
> > +
> > + print(f"Can't parse version {version}")
> > + return (0, 0, 0)
> > +
> > + @staticmethod
> > + def find_python(min_version):
> > + """
> > + Detect if are out there any python 3.xy version newer than the
> > + current one.
> > +
> > + Note: this routine is limited to up to 2 digits for python3. We
> > + may need to update it one day, hopefully on a distant future.
> > + """
> > + patterns = [
> > + "python3.[0-9]",
> > + "python3.[0-9][0-9]",
> > + ]
> > +
> > + # Seek for a python binary newer than min_version
> > + for path in os.getenv("PATH", "").split(":"):
> > + for pattern in patterns:
> > + for cmd in glob(os.path.join(path, pattern)):
> > + if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
> > + version = PythonVersion.get_python_version(cmd)
> > + if version >= min_version:
> > + return cmd
> > +
> > + return None
> > +
> > + @staticmethod
> > + def check_python(min_version):
> > + """
> > + Check if the current python binary satisfies our minimal requirement
> > + for Sphinx build. If not, re-run with a newer version if found.
> > + """
> > + cur_ver = sys.version_info[:3]
> > + if cur_ver >= min_version:
> > + ver = PythonVersion.ver_str(cur_ver)
> > + print(f"Python version: {ver}")
> > +
> > + return
> > +
> > + python_ver = PythonVersion.ver_str(cur_ver)
> > +
> > + new_python_cmd = PythonVersion.find_python(min_version)
> > + if not new_python_cmd:
> > + print(f"ERROR: Python version {python_ver} is not spported anymore\n")
> > + print(" Can't find a new version. This script may fail")
> > + return
> > +
> > + # Restart script using the newer version
>
> I thought the whole idea of restarting was completely rejected by
> approximately everyone?!
This patch is just moving the code. There is a patch after this one
changing the behavior.
> BR,
> Jani.
>
>
> > + script_path = os.path.abspath(sys.argv[0])
> > + args = [new_python_cmd, script_path] + sys.argv[1:]
> > +
> > + print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
> > +
> > + try:
> > + os.execv(new_python_cmd, args)
> > + except OSError as e:
> > + sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
> > diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install
> > index 954ed3dc0645..d6d673b7945c 100755
> > --- a/tools/docs/sphinx-pre-install
> > +++ b/tools/docs/sphinx-pre-install
> > @@ -32,20 +32,10 @@ import subprocess
> > import sys
> > from glob import glob
> >
> > +from lib.python_version import PythonVersion
> >
> > -def parse_version(version):
> > - """Convert a major.minor.patch version into a tuple"""
> > - return tuple(int(x) for x in version.split("."))
> > -
> > -
> > -def ver_str(version):
> > - """Returns a version tuple as major.minor.patch"""
> > -
> > - return ".".join([str(x) for x in version])
> > -
> > -
> > -RECOMMENDED_VERSION = parse_version("3.4.3")
> > -MIN_PYTHON_VERSION = parse_version("3.7")
> > +RECOMMENDED_VERSION = PythonVersion("3.4.3").version
> > +MIN_PYTHON_VERSION = PythonVersion("3.7").version
> >
> >
> > class DepManager:
> > @@ -235,95 +225,11 @@ class AncillaryMethods:
> >
> > return None
> >
> > - @staticmethod
> > - def get_python_version(cmd):
> > - """
> > - Get python version from a Python binary. As we need to detect if
> > - are out there newer python binaries, we can't rely on sys.release here.
> > - """
> > -
> > - result = SphinxDependencyChecker.run([cmd, "--version"],
> > - capture_output=True, text=True)
> > - version = result.stdout.strip()
> > -
> > - match = re.search(r"(\d+\.\d+\.\d+)", version)
> > - if match:
> > - return parse_version(match.group(1))
> > -
> > - print(f"Can't parse version {version}")
> > - return (0, 0, 0)
> > -
> > - @staticmethod
> > - def find_python():
> > - """
> > - Detect if are out there any python 3.xy version newer than the
> > - current one.
> > -
> > - Note: this routine is limited to up to 2 digits for python3. We
> > - may need to update it one day, hopefully on a distant future.
> > - """
> > - patterns = [
> > - "python3.[0-9]",
> > - "python3.[0-9][0-9]",
> > - ]
> > -
> > - # Seek for a python binary newer than MIN_PYTHON_VERSION
> > - for path in os.getenv("PATH", "").split(":"):
> > - for pattern in patterns:
> > - for cmd in glob(os.path.join(path, pattern)):
> > - if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
> > - version = SphinxDependencyChecker.get_python_version(cmd)
> > - if version >= MIN_PYTHON_VERSION:
> > - return cmd
> > -
> > - @staticmethod
> > - def check_python():
> > - """
> > - Check if the current python binary satisfies our minimal requirement
> > - for Sphinx build. If not, re-run with a newer version if found.
> > - """
> > - cur_ver = sys.version_info[:3]
> > - if cur_ver >= MIN_PYTHON_VERSION:
> > - ver = ver_str(cur_ver)
> > - print(f"Python version: {ver}")
> > -
> > - # This could be useful for debugging purposes
> > - if SphinxDependencyChecker.which("docutils"):
> > - result = SphinxDependencyChecker.run(["docutils", "--version"],
> > - capture_output=True, text=True)
> > - ver = result.stdout.strip()
> > - match = re.search(r"(\d+\.\d+\.\d+)", ver)
> > - if match:
> > - ver = match.group(1)
> > -
> > - print(f"Docutils version: {ver}")
> > -
> > - return
> > -
> > - python_ver = ver_str(cur_ver)
> > -
> > - new_python_cmd = SphinxDependencyChecker.find_python()
> > - if not new_python_cmd:
> > - print(f"ERROR: Python version {python_ver} is not spported anymore\n")
> > - print(" Can't find a new version. This script may fail")
> > - return
> > -
> > - # Restart script using the newer version
> > - script_path = os.path.abspath(sys.argv[0])
> > - args = [new_python_cmd, script_path] + sys.argv[1:]
> > -
> > - print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
> > -
> > - try:
> > - os.execv(new_python_cmd, args)
> > - except OSError as e:
> > - sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
> > -
> > @staticmethod
> > def run(*args, **kwargs):
> > """
> > Excecute a command, hiding its output by default.
> > - Preserve comatibility with older Python versions.
> > + Preserve compatibility with older Python versions.
> > """
> >
> > capture_output = kwargs.pop('capture_output', False)
> > @@ -527,11 +433,11 @@ class MissingCheckers(AncillaryMethods):
> > for line in result.stdout.split("\n"):
> > match = re.match(r"^sphinx-build\s+([\d\.]+)(?:\+(?:/[\da-f]+)|b\d+)?\s*$", line)
> > if match:
> > - return parse_version(match.group(1))
> > + return PythonVersion.parse_version(match.group(1))
> >
> > match = re.match(r"^Sphinx.*\s+([\d\.]+)\s*$", line)
> > if match:
> > - return parse_version(match.group(1))
> > + return PythonVersion.parse_version(match.group(1))
> >
> > def check_sphinx(self, conf):
> > """
> > @@ -542,7 +448,7 @@ class MissingCheckers(AncillaryMethods):
> > for line in f:
> > match = re.match(r"^\s*needs_sphinx\s*=\s*[\'\"]([\d\.]+)[\'\"]", line)
> > if match:
> > - self.min_version = parse_version(match.group(1))
> > + self.min_version = PythonVersion.parse_version(match.group(1))
> > break
> > except IOError:
> > sys.exit(f"Can't open {conf}")
> > @@ -562,8 +468,8 @@ class MissingCheckers(AncillaryMethods):
> > sys.exit(f"{sphinx} didn't return its version")
> >
> > if self.cur_version < self.min_version:
> > - curver = ver_str(self.cur_version)
> > - minver = ver_str(self.min_version)
> > + curver = PythonVersion.ver_str(self.cur_version)
> > + minver = PythonVersion.ver_str(self.min_version)
> >
> > print(f"ERROR: Sphinx version is {curver}. It should be >= {minver}")
> > self.need_sphinx = 1
> > @@ -1304,7 +1210,7 @@ class SphinxDependencyChecker(MissingCheckers):
> > else:
> > if self.need_sphinx and ver >= self.min_version:
> > return (f, ver)
> > - elif parse_version(ver) > self.cur_version:
> > + elif PythonVersion.parse_version(ver) > self.cur_version:
> > return (f, ver)
> >
> > return ("", ver)
> > @@ -1411,7 +1317,7 @@ class SphinxDependencyChecker(MissingCheckers):
> > return
> >
> > if self.latest_avail_ver:
> > - latest_avail_ver = ver_str(self.latest_avail_ver)
> > + latest_avail_ver = PythonVersion.ver_str(self.latest_avail_ver)
> >
> > if not self.need_sphinx:
> > # sphinx-build is present and its version is >= $min_version
> > @@ -1507,7 +1413,7 @@ class SphinxDependencyChecker(MissingCheckers):
> > else:
> > print("Unknown OS")
> > if self.cur_version != (0, 0, 0):
> > - ver = ver_str(self.cur_version)
> > + ver = PythonVersion.ver_str(self.cur_version)
> > print(f"Sphinx version: {ver}\n")
> >
> > # Check the type of virtual env, depending on Python version
> > @@ -1613,7 +1519,7 @@ def main():
> >
> > checker = SphinxDependencyChecker(args)
> >
> > - checker.check_python()
> > + PythonVersion.check_python(MIN_PYTHON_VERSION)
> > checker.check_needs()
> >
> > # Call main if not used as module
>
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install
2025-09-10 12:24 ` Mauro Carvalho Chehab
@ 2025-09-11 10:28 ` Jani Nikula
2025-09-11 10:45 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-11 10:28 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Jonathan Corbet, Linux Doc Mailing List, linux-kernel
On Wed, 10 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Wed, 10 Sep 2025 13:14:33 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
>> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> > + # Restart script using the newer version
>>
>> I thought the whole idea of restarting was completely rejected by
>> approximately everyone?!
>
> This patch is just moving the code. There is a patch after this one
> changing the behavior.
Shouldn't we fix that kind of problems first, instead of refactoring?
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install
2025-09-11 10:28 ` Jani Nikula
@ 2025-09-11 10:45 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-11 10:45 UTC (permalink / raw)
To: Jani Nikula
Cc: Mauro Carvalho Chehab, Jonathan Corbet, Linux Doc Mailing List,
linux-kernel
On Thu, Sep 11, 2025 at 01:28:28PM +0300, Jani Nikula wrote:
> On Wed, 10 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Em Wed, 10 Sep 2025 13:14:33 +0300
> > Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> >
> >> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >> > + # Restart script using the newer version
> >>
> >> I thought the whole idea of restarting was completely rejected by
> >> approximately everyone?!
> >
> > This patch is just moving the code. There is a patch after this one
> > changing the behavior.
>
> Shouldn't we fix that kind of problems first, instead of refactoring?
The patch order doesn't matter, provided that it won't break bisect.
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* [PATCH v4 06/19] tools/docs: python_version: drop a debug print
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (4 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 05/19] tools/docs: python_version: move version check from sphinx-pre-install Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 07/19] tools/docs: python_version: allow check for alternatives and bail out Mauro Carvalho Chehab
` (13 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
The version print at the lib was added for debugging purposes.
Get rid of it.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/lib/python_version.py | 2 --
1 file changed, 2 deletions(-)
diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_version.py
index 0519d524e547..660bfe7d23fa 100644
--- a/tools/docs/lib/python_version.py
+++ b/tools/docs/lib/python_version.py
@@ -109,8 +109,6 @@ class PythonVersion:
cur_ver = sys.version_info[:3]
if cur_ver >= min_version:
ver = PythonVersion.ver_str(cur_ver)
- print(f"Python version: {ver}")
-
return
python_ver = PythonVersion.ver_str(cur_ver)
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 07/19] tools/docs: python_version: allow check for alternatives and bail out
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (5 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 06/19] tools/docs: python_version: drop a debug print Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
` (12 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
The caller script may not want an automatic execution of the new
version. Add two parameters to allow showing alternatives and to
bail out if version is incompatible.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/lib/python_version.py | 43 ++++++++++++++++++++++++++------
1 file changed, 35 insertions(+), 8 deletions(-)
diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_version.py
index 660bfe7d23fa..a9fda2470a26 100644
--- a/tools/docs/lib/python_version.py
+++ b/tools/docs/lib/python_version.py
@@ -85,10 +85,12 @@ class PythonVersion:
may need to update it one day, hopefully on a distant future.
"""
patterns = [
- "python3.[0-9]",
"python3.[0-9][0-9]",
+ "python3.[0-9]",
]
+ python_cmd = []
+
# Seek for a python binary newer than min_version
for path in os.getenv("PATH", "").split(":"):
for pattern in patterns:
@@ -96,12 +98,13 @@ class PythonVersion:
if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
version = PythonVersion.get_python_version(cmd)
if version >= min_version:
- return cmd
+ python_cmd.append((version, cmd))
- return None
+ return sorted(python_cmd, reverse=True)
@staticmethod
- def check_python(min_version):
+ def check_python(min_version, show_alternatives=False, bail_out=False,
+ success_on_error=False):
"""
Check if the current python binary satisfies our minimal requirement
for Sphinx build. If not, re-run with a newer version if found.
@@ -113,18 +116,42 @@ class PythonVersion:
python_ver = PythonVersion.ver_str(cur_ver)
- new_python_cmd = PythonVersion.find_python(min_version)
- if not new_python_cmd:
+ available_versions = PythonVersion.find_python(min_version)
+ if not available_versions:
print(f"ERROR: Python version {python_ver} is not spported anymore\n")
print(" Can't find a new version. This script may fail")
return
- # Restart script using the newer version
script_path = os.path.abspath(sys.argv[0])
- args = [new_python_cmd, script_path] + sys.argv[1:]
+
+ # Check possible alternatives
+ if available_versions:
+ new_python_cmd = available_versions[0][1]
+ else:
+ new_python_cmd = None
+
+ if show_alternatives:
+ print("You could run, instead:")
+ for _, cmd in available_versions:
+ args = [cmd, script_path] + sys.argv[1:]
+
+ cmd_str = " ".join(args)
+ print(f" {cmd_str}")
+ print()
+
+ if bail_out:
+ msg = f"Python {python_ver} not supported. Bailing out"
+ if success_on_error:
+ print(msg, file=sys.stderr)
+ sys.exit(0)
+ else:
+ sys.exit(msg)
print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
+ # Restart script using the newer version
+ args = [new_python_cmd, script_path] + sys.argv[1:]
+
try:
os.execv(new_python_cmd, args)
except OSError as e:
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (6 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 07/19] tools/docs: python_version: allow check for alternatives and bail out Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-09 14:53 ` Jonathan Corbet
` (2 more replies)
2025-09-04 7:33 ` [PATCH v4 09/19] tools/docs: sphinx-build-wrapper: add comments and blank lines Mauro Carvalho Chehab
` (11 subsequent siblings)
19 siblings, 3 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Andreas Hindborg, Benno Lossin, Boqun Feng,
Danilo Krummrich, Gary Guo, Mauro Carvalho Chehab, Miguel Ojeda,
Trevor Gross, linux-kernel, rust-for-linux
There are too much magic inside docs Makefile to properly run
sphinx-build. Create an ancillary script that contains all
kernel-related sphinx-build call logic currently at Makefile.
Such script is designed to work both as an standalone command
and as part of a Makefile. As such, it properly handles POSIX
jobserver used by GNU make.
On a side note, there was a line number increase due to the
conversion:
Documentation/Makefile | 131 +++----------
tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++
2 files changed, 323 insertions(+), 101 deletions(-)
This is because some things are more verbosed on Python and because
it requires reading env vars from Makefile. Besides it, this script
has some extra features that don't exist at the Makefile:
- It can be called directly from command line;
- It properly return PDF build errors.
When running the script alone, it will only take handle sphinx-build
targets. On other words, it won't runn make rustdoc after building
htmlfiles, nor it will run the extra check scripts.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 131 ++++----------
tools/docs/sphinx-build-wrapper | 293 ++++++++++++++++++++++++++++++++
2 files changed, 323 insertions(+), 101 deletions(-)
create mode 100755 tools/docs/sphinx-build-wrapper
diff --git a/Documentation/Makefile b/Documentation/Makefile
index deb2029228ed..4736f02b6c9e 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -23,21 +23,22 @@ SPHINXOPTS =
SPHINXDIRS = .
DOCS_THEME =
DOCS_CSS =
-_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
SPHINX_CONF = conf.py
PAPER =
BUILDDIR = $(obj)/output
PDFLATEX = xelatex
LATEXOPTS = -interaction=batchmode -no-shell-escape
+PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
+
+# Wrapper for sphinx-build
+
+BUILD_WRAPPER = $(srctree)/tools/docs/sphinx-build-wrapper
+
# For denylisting "variable font" files
# Can be overridden by setting as an env variable
FONTS_CONF_DENY_VF ?= $(HOME)/deny-vf
-ifeq ($(findstring 1, $(KBUILD_VERBOSE)),)
-SPHINXOPTS += "-q"
-endif
-
# User-friendly check for sphinx-build
HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)
@@ -51,63 +52,31 @@ ifeq ($(HAVE_SPHINX),0)
else # HAVE_SPHINX
-# User-friendly check for pdflatex and latexmk
-HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
-HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
+# Common documentation targets
+infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
+ $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
+ +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
+ --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
+ --builddir="$(BUILDDIR)" \
+ --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
-ifeq ($(HAVE_LATEXMK),1)
- PDFLATEX := latexmk -$(PDFLATEX)
-endif #HAVE_LATEXMK
-
-# Internal variables.
-PAPEROPT_a4 = -D latex_elements.papersize=a4paper
-PAPEROPT_letter = -D latex_elements.papersize=letterpaper
-ALLSPHINXOPTS = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
-ALLSPHINXOPTS += $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
-ifneq ($(wildcard $(srctree)/.config),)
-ifeq ($(CONFIG_RUST),y)
- # Let Sphinx know we will include rustdoc
- ALLSPHINXOPTS += -t rustdoc
-endif
+# Special handling for pdfdocs
+ifeq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0)
+pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
+else
+pdfdocs:
+ $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
+ @echo " SKIP Sphinx $@ target."
endif
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-
-# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
-loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
-
-# $2 sphinx builder e.g. "html"
-# $3 name of the build subfolder / e.g. "userspace-api/media", used as:
-# * dest folder relative to $(BUILDDIR) and
-# * cache folder relative to $(BUILDDIR)/.doctrees
-# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man
-# $5 reST source folder relative to $(src),
-# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentation/userspace-api/media
-
-PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
-
-quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
- cmd_sphinx = \
- PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
- BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \
- $(PYTHON3) $(srctree)/scripts/jobserver-exec \
- $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
- $(SPHINXBUILD) \
- -b $2 \
- -c $(abspath $(src)) \
- -d $(abspath $(BUILDDIR)/.doctrees/$3) \
- -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
- $(ALLSPHINXOPTS) \
- $(abspath $(src)/$5) \
- $(abspath $(BUILDDIR)/$3/$4) && \
- if [ "x$(DOCS_CSS)" != "x" ]; then \
- cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(DOCS_CSS)) $(BUILDDIR)/$3/_static/; \
- fi
+# HTML main logic is identical to other targets. However, if rust is enabled,
+# an extra step at the end is required to generate rustdoc.
htmldocs:
- @$(srctree)/tools/docs/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
-
+ $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
+ +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
+ --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
+ --builddir="$(BUILDDIR)" \
+ --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
# If Rust support is available and .config exists, add rustdoc generated contents.
# If there are any, the errors from this make rustdoc will be displayed but
# won't stop the execution of htmldocs
@@ -118,49 +87,6 @@ ifeq ($(CONFIG_RUST),y)
endif
endif
-texinfodocs:
- @$(srctree)/tools/docs/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
-
-# Note: the 'info' Make target is generated by sphinx itself when
-# running the texinfodocs target define above.
-infodocs: texinfodocs
- $(MAKE) -C $(BUILDDIR)/texinfo info
-
-linkcheckdocs:
- @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
-
-latexdocs:
- @$(srctree)/tools/docs/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
-
-ifeq ($(HAVE_PDFLATEX),0)
-
-pdfdocs:
- $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
- @echo " SKIP Sphinx $@ target."
-
-else # HAVE_PDFLATEX
-
-pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
-pdfdocs: latexdocs
- @$(srctree)/tools/docs/sphinx-pre-install --version-check
- $(foreach var,$(SPHINXDIRS), \
- $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
- mkdir -p $(BUILDDIR)/$(var)/pdf; \
- mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
- )
-
-endif # HAVE_PDFLATEX
-
-epubdocs:
- @$(srctree)/tools/docs/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
-
-xmldocs:
- @$(srctree)/tools/docs/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
-
endif # HAVE_SPHINX
# The following targets are independent of HAVE_SPHINX, and the rules should
@@ -172,6 +98,9 @@ refcheckdocs:
cleandocs:
$(Q)rm -rf $(BUILDDIR)
+# Used only on help
+_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
+
dochelp:
@echo ' Linux kernel internal documentation in different formats from ReST:'
@echo ' htmldocs - HTML'
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
new file mode 100755
index 000000000000..3256418d8dc5
--- /dev/null
+++ b/tools/docs/sphinx-build-wrapper
@@ -0,0 +1,293 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+import argparse
+import os
+import shlex
+import shutil
+import subprocess
+import sys
+from lib.python_version import PythonVersion
+
+LIB_DIR = "../../scripts/lib"
+SRC_DIR = os.path.dirname(os.path.realpath(__file__))
+sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
+
+from jobserver import JobserverExec
+
+MIN_PYTHON_VERSION = PythonVersion("3.7").version
+PAPER = ["", "a4", "letter"]
+TARGETS = {
+ "cleandocs": { "builder": "clean" },
+ "linkcheckdocs": { "builder": "linkcheck" },
+ "htmldocs": { "builder": "html" },
+ "epubdocs": { "builder": "epub", "out_dir": "epub" },
+ "texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" },
+ "infodocs": { "builder": "texinfo", "out_dir": "texinfo" },
+ "latexdocs": { "builder": "latex", "out_dir": "latex" },
+ "pdfdocs": { "builder": "latex", "out_dir": "latex" },
+ "xmldocs": { "builder": "xml", "out_dir": "xml" },
+}
+
+class SphinxBuilder:
+ def is_rust_enabled(self):
+ config_path = os.path.join(self.srctree, ".config")
+ if os.path.isfile(config_path):
+ with open(config_path, "r", encoding="utf-8") as f:
+ return "CONFIG_RUST=y" in f.read()
+ return False
+
+ def get_path(self, path, use_cwd=False, abs_path=False):
+ path = os.path.expanduser(path)
+ if not path.startswith("/"):
+ if use_cwd:
+ base = os.getcwd()
+ else:
+ base = self.srctree
+ path = os.path.join(base, path)
+ if abs_path:
+ return os.path.abspath(path)
+ return path
+
+ def __init__(self, builddir, verbose=False, n_jobs=None):
+ self.verbose = None
+ self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
+ self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
+ self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
+ self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+ if not verbose:
+ verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
+ if verbose is not None:
+ self.verbose = verbose
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-j', '--jobs', type=int)
+ parser.add_argument('-q', '--quiet', type=int)
+ sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", ""))
+ sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
+ if sphinx_args.quiet is True:
+ self.verbose = False
+ if sphinx_args.jobs:
+ self.n_jobs = sphinx_args.jobs
+ self.n_jobs = n_jobs
+ self.srctree = os.environ.get("srctree")
+ if not self.srctree:
+ self.srctree = "."
+ os.environ["srctree"] = self.srctree
+ self.sphinxbuild = os.environ.get("SPHINXBUILD", "sphinx-build")
+ self.kerneldoc = self.get_path(os.environ.get("KERNELDOC",
+ "scripts/kernel-doc.py"))
+ self.builddir = self.get_path(builddir, use_cwd=True, abs_path=True)
+
+ self.config_rust = self.is_rust_enabled()
+
+ self.pdflatex_cmd = shutil.which(self.pdflatex)
+ self.latexmk_cmd = shutil.which("latexmk")
+
+ self.env = os.environ.copy()
+
+ def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
+ with JobserverExec() as jobserver:
+ if jobserver.claim:
+ n_jobs = str(jobserver.claim)
+ else:
+ n_jobs = "auto" # Supported since Sphinx 1.7
+ cmd = []
+ cmd.append(sys.executable)
+ cmd.append(sphinx_build)
+ if self.n_jobs:
+ n_jobs = str(self.n_jobs)
+
+ if n_jobs:
+ cmd += [f"-j{n_jobs}"]
+
+ if not self.verbose:
+ cmd.append("-q")
+ cmd += self.sphinxopts
+ cmd += build_args
+ if self.verbose:
+ print(" ".join(cmd))
+ return subprocess.call(cmd, *args, **pwargs)
+
+ def handle_html(self, css, output_dir):
+ if not css:
+ return
+ css = os.path.expanduser(css)
+ if not css.startswith("/"):
+ css = os.path.join(self.srctree, css)
+ static_dir = os.path.join(output_dir, "_static")
+ os.makedirs(static_dir, exist_ok=True)
+ try:
+ shutil.copy2(css, static_dir)
+ except (OSError, IOError) as e:
+ print(f"Warning: Failed to copy CSS: {e}", file=sys.stderr)
+
+ def handle_pdf(self, output_dirs):
+ builds = {}
+ max_len = 0
+ for from_dir in output_dirs:
+ pdf_dir = os.path.join(from_dir, "../pdf")
+ os.makedirs(pdf_dir, exist_ok=True)
+ if self.latexmk_cmd:
+ latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
+ else:
+ latex_cmd = [self.pdflatex]
+ latex_cmd.extend(shlex.split(self.latexopts))
+ tex_suffix = ".tex"
+ has_tex = False
+ build_failed = False
+ with os.scandir(from_dir) as it:
+ for entry in it:
+ if not entry.name.endswith(tex_suffix):
+ continue
+ name = entry.name[:-len(tex_suffix)]
+ has_tex = True
+ try:
+ subprocess.run(latex_cmd + [entry.path],
+ cwd=from_dir, check=True)
+ except subprocess.CalledProcessError:
+ pass
+ pdf_name = name + ".pdf"
+ pdf_from = os.path.join(from_dir, pdf_name)
+ pdf_to = os.path.join(pdf_dir, pdf_name)
+ if os.path.exists(pdf_from):
+ os.rename(pdf_from, pdf_to)
+ builds[name] = os.path.relpath(pdf_to, self.builddir)
+ else:
+ builds[name] = "FAILED"
+ build_failed = True
+ name = entry.name.removesuffix(".tex")
+ max_len = max(max_len, len(name))
+
+ if not has_tex:
+ name = os.path.basename(from_dir)
+ max_len = max(max_len, len(name))
+ builds[name] = "FAILED (no .tex)"
+ build_failed = True
+ msg = "Summary"
+ msg += "\n" + "=" * len(msg)
+ print()
+ print(msg)
+ for pdf_name, pdf_file in builds.items():
+ print(f"{pdf_name:<{max_len}}: {pdf_file}")
+ print()
+ if build_failed:
+ sys.exit("PDF build failed: not all PDF files were created.")
+ else:
+ print("All PDF files were built.")
+
+ def handle_info(self, output_dirs):
+ for output_dir in output_dirs:
+ try:
+ subprocess.run(["make", "info"], cwd=output_dir, check=True)
+ except subprocess.CalledProcessError as e:
+ sys.exit(f"Error generating info docs: {e}")
+
+ def cleandocs(self, builder):
+ shutil.rmtree(self.builddir, ignore_errors=True)
+
+ def build(self, target, sphinxdirs=None, conf="conf.py",
+ theme=None, css=None, paper=None):
+ builder = TARGETS[target]["builder"]
+ out_dir = TARGETS[target].get("out_dir", "")
+ if target == "cleandocs":
+ self.cleandocs(builder)
+ return
+ if theme:
+ os.environ["DOCS_THEME"] = theme
+ sphinxbuild = shutil.which(self.sphinxbuild, path=self.env["PATH"])
+ if not sphinxbuild:
+ sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n")
+ if builder == "latex":
+ if not self.pdflatex_cmd and not self.latexmk_cmd:
+ sys.exit("Error: pdflatex or latexmk required for PDF generation")
+ docs_dir = os.path.abspath(os.path.join(self.srctree, "Documentation"))
+ kerneldoc = self.kerneldoc
+ if kerneldoc.startswith(self.srctree):
+ kerneldoc = os.path.relpath(kerneldoc, self.srctree)
+ args = [ "-b", builder, "-c", docs_dir ]
+ if builder == "latex":
+ if not paper:
+ paper = PAPER[1]
+ args.extend(["-D", f"latex_elements.papersize={paper}paper"])
+ if self.config_rust:
+ args.extend(["-t", "rustdoc"])
+ if conf:
+ self.env["SPHINX_CONF"] = self.get_path(conf, abs_path=True)
+ if not sphinxdirs:
+ sphinxdirs = os.environ.get("SPHINXDIRS", ".")
+ sphinxdirs_list = []
+ for sphinxdir in sphinxdirs:
+ if isinstance(sphinxdir, list):
+ sphinxdirs_list += sphinxdir
+ else:
+ for name in sphinxdir.split(" "):
+ sphinxdirs_list.append(name)
+ output_dirs = []
+ for sphinxdir in sphinxdirs_list:
+ src_dir = os.path.join(docs_dir, sphinxdir)
+ doctree_dir = os.path.join(self.builddir, ".doctrees")
+ output_dir = os.path.join(self.builddir, sphinxdir, out_dir)
+ src_dir = os.path.normpath(src_dir)
+ doctree_dir = os.path.normpath(doctree_dir)
+ output_dir = os.path.normpath(output_dir)
+ os.makedirs(doctree_dir, exist_ok=True)
+ os.makedirs(output_dir, exist_ok=True)
+ output_dirs.append(output_dir)
+ build_args = args + [
+ "-d", doctree_dir,
+ "-D", f"kerneldoc_bin={kerneldoc}",
+ "-D", f"version={self.kernelversion}",
+ "-D", f"release={self.kernelrelease}",
+ "-D", f"kerneldoc_srctree={self.srctree}",
+ src_dir,
+ output_dir,
+ ]
+ try:
+ self.run_sphinx(sphinxbuild, build_args, env=self.env)
+ except (OSError, ValueError, subprocess.SubprocessError) as e:
+ sys.exit(f"Build failed: {repr(e)}")
+ if target in ["htmldocs", "epubdocs"]:
+ self.handle_html(css, output_dir)
+ if target == "pdfdocs":
+ self.handle_pdf(output_dirs)
+ elif target == "infodocs":
+ self.handle_info(output_dirs)
+
+def jobs_type(value):
+ if value is None:
+ return None
+ if value.lower() == 'auto':
+ return value.lower()
+ try:
+ if int(value) >= 1:
+ return value
+ raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}")
+ except ValueError:
+ raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}")
+
+def main():
+ parser = argparse.ArgumentParser(description="Kernel documentation builder")
+ parser.add_argument("target", choices=list(TARGETS.keys()),
+ help="Documentation target to build")
+ parser.add_argument("--sphinxdirs", nargs="+",
+ help="Specific directories to build")
+ parser.add_argument("--conf", default="conf.py",
+ help="Sphinx configuration file")
+ parser.add_argument("--builddir", default="output",
+ help="Sphinx configuration file")
+ parser.add_argument("--theme", help="Sphinx theme to use")
+ parser.add_argument("--css", help="Custom CSS file for HTML/EPUB")
+ parser.add_argument("--paper", choices=PAPER, default=PAPER[0],
+ help="Paper size for LaTeX/PDF output")
+ parser.add_argument("-v", "--verbose", action='store_true',
+ help="place build in verbose mode")
+ parser.add_argument('-j', '--jobs', type=jobs_type,
+ help="Sets number of jobs to use with sphinx-build")
+ args = parser.parse_args()
+ PythonVersion.check_python(MIN_PYTHON_VERSION)
+ builder = SphinxBuilder(builddir=args.builddir,
+ verbose=args.verbose, n_jobs=args.jobs)
+ builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
+ theme=args.theme, css=args.css, paper=args.paper)
+
+if __name__ == "__main__":
+ main()
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-04 7:33 ` [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
@ 2025-09-09 14:53 ` Jonathan Corbet
2025-09-09 15:59 ` Mauro Carvalho Chehab
2025-09-09 15:21 ` Jonathan Corbet
2025-09-10 10:46 ` Jani Nikula
2 siblings, 1 reply; 67+ messages in thread
From: Jonathan Corbet @ 2025-09-09 14:53 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Andreas Hindborg, Benno Lossin, Boqun Feng,
Danilo Krummrich, Gary Guo, Mauro Carvalho Chehab, Miguel Ojeda,
Trevor Gross, linux-kernel, rust-for-linux
Finally beginning to look at this. I'm working from the pulled version,
rather than the commentless patch (please don't do that again :). A nit
from SphinxBuilder::__init__():
> #
> # As we handle number of jobs and quiet in separate, we need to pick
> # both the same way as sphinx-build would pick, optionally accepts
> # whitespaces or not. So let's use argparse to handle argument expansion
> #
> parser = argparse.ArgumentParser()
> parser.add_argument('-j', '--jobs', type=int)
> parser.add_argument('-q', '--quiet', type=int)
>
> #
> # Other sphinx-build arguments go as-is, so place them
> # at self.sphinxopts, using shell parser
> #
> sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", "))
>
> #
> # Build a list of sphinx args
> #
> sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
> if sphinx_args.quiet is True:
> self.verbose = False
>
> if sphinx_args.jobs:
> self.n_jobs = sphinx_args.jobs
>
> #
> # If the command line argument "-j" is used override SPHINXOPTS
> #
>
> self.n_jobs = n_jobs
First of all, I do wish you would isolate this sort of concern into its
own function. But, beyond that, you go to all that effort to parse the
--jobs flag, but that last line just throws it all away. What was the
real purpose here?
Thanks,
jon
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-09 14:53 ` Jonathan Corbet
@ 2025-09-09 15:59 ` Mauro Carvalho Chehab
2025-09-09 18:56 ` Jonathan Corbet
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-09 15:59 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
On Tue, Sep 09, 2025 at 08:53:50AM -0600, Jonathan Corbet wrote:
> Finally beginning to look at this. I'm working from the pulled version,
> rather than the commentless patch (please don't do that again :).
Heh, when I had to rebase it, I noticed it was a bad idea to split ;-)
I'll merge the commentless patch at the next respin.
> A nit
> from SphinxBuilder::__init__():
>
> > #
> > # As we handle number of jobs and quiet in separate, we need to pick
> > # both the same way as sphinx-build would pick, optionally accepts
> > # whitespaces or not. So let's use argparse to handle argument expansion
> > #
> > parser = argparse.ArgumentParser()
> > parser.add_argument('-j', '--jobs', type=int)
> > parser.add_argument('-q', '--quiet', type=int)
> >
> > #
> > # Other sphinx-build arguments go as-is, so place them
> > # at self.sphinxopts, using shell parser
> > #
> > sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", "))
> >
> > #
> > # Build a list of sphinx args
> > #
> > sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
> > if sphinx_args.quiet is True:
> > self.verbose = False
> >
> > if sphinx_args.jobs:
> > self.n_jobs = sphinx_args.jobs
> >
> > #
> > # If the command line argument "-j" is used override SPHINXOPTS
> > #
> >
> > self.n_jobs = n_jobs
>
> First of all, I do wish you would isolate this sort of concern into its
> own function.
Ok.
> But, beyond that, you go to all that effort to parse the
> --jobs flag, but that last line just throws it all away. What was the
> real purpose here?
heh, it sounds to be something that got lost during a rebase.
This should be, instead:
if n_jobs:
self.n_jobs = n_jobs # this is parser.parse_args().n_jobs from main()
-
Basically, what happens is that the number of jobs can be on
different places:
1) if called via Makefile, no job arguments are passed at
command line, but SPHINXOPTS may contain "-j" on it.
The code shall use jobserver to get it by default, with:
# Clain all remaining jobs from make jobserver pool
with JobserverExec() as jobserver:
if jobserver.claim:
n_jobs = str(jobserver.claim)
else:
n_jobs = "auto"
# some logic to call sphinx-build with a parallel flag
# After with, claim is returned back to the
# jobserver, to allow other jobs to be executed
# in parallel, if any.
this basically claims all remaining make jobs from GNU jobserver.
So, if the build started with "-j8" and make was called with
other args, the number of available slots could be, for
instance "4".
The above logic will have jobserver.claim = 4, and run:
sphinx-build -j4 <other args>
This is the normal behavior when one does, for instance:
make -j8 drivers/media htmldocs
2) if called with SPHINXOPTS="-j8", it shall ignore jobserver
and call sphinx-build with -j8;
both cases (1) and (2) are handler inside a function
-
Now, when sphinx-build-wrapper is called from command line,
there's no GNU jobserver. So:
3) by default, it uses "-jauto". This can be problematic on
machines with a large number of CPUs but without too much
free memory (with Sphinx 7.x, one needs a really huge amount
of RAM to run sphinx with -j - like 128GB or more with -j24)
4) if "-j" parameter is specified, pass it as-is to sphinx-build;
tools/docs/sphinx-build-wrapper -j16 htmldocs
this calls sphinx-build with -j16.
5) one might still use:
SPHINXOPTS=-j8 tools/docs/sphinx-build-wrapper htmldocs
or, even weirder:
SPHINXOPTS=-j8 tools/docs/sphinx-build-wrapper -j16 htmldocs
The above logic you reviewed is handling (4) and (5). There:
- n_jobs comes from command line;
- this comes from SPHINXOPTS var:
sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", ""))
if both SPHINXOPTS and -j are specified like:
SPHINXOPTS=-j8 tools/docs/sphinx-build-wrapper -j16 htmldocs
IMO it shall pick the latest one (-j16).
Yet, perhaps I should have written the code on a different way,
e.g., like:
if n_jobs:
# Command line argument takes precedence
self.n_jobs = n_jobs
elif sphinx_args.jobs:
# Otherwise, use what it was specified at SPHINXOPTS if
# any
self.n_jobs = sphinx_args.jobs
I'll change it at the next spin and re-test it for all 5 scenarios.
Regards,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-09 15:59 ` Mauro Carvalho Chehab
@ 2025-09-09 18:56 ` Jonathan Corbet
2025-09-09 20:53 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jonathan Corbet @ 2025-09-09 18:56 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> Basically, what happens is that the number of jobs can be on
> different places:
There is a lot of complexity there, and spread out between __init__(),
run_sphinx(), and handle_pdf(). Is there any way to create a single
figure_out_how_many_damn_jobs() and coalesce that logic there? That
would help make that part of the system a bit more comprehensible.
That said, I've been unable to make this change break in my testing. I
guess I'm not seeing a lot of impediments to applying the next version
at this point.
Thanks,
jon
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-09 18:56 ` Jonathan Corbet
@ 2025-09-09 20:53 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-09 20:53 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Linux Doc Mailing List, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross, linux-kernel,
rust-for-linux
Em Tue, 09 Sep 2025 12:56:17 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
> > Basically, what happens is that the number of jobs can be on
> > different places:
>
> There is a lot of complexity there, and spread out between __init__(),
> run_sphinx(), and handle_pdf(). Is there any way to create a single
> figure_out_how_many_damn_jobs() and coalesce that logic there? That
> would help make that part of the system a bit more comprehensible.
I'll try to better organize it, but run_sphinx() does something
different than handle_pdf():
- run_sphinx: claims all tokens;
- handle_pdf: use future.concurrent and handle parallelism inside it.
Perhaps I can move the future.concurrent parallelism to jobserver library
to simplify the code a little bit while offering an interface somewhat similar
to run_sphinx logic. Let's see if I can find a way to do it while keeping
the code generic (*).
Will take a look on it probably on Thursday of Friday.
(*) I did one similar attempt at devel time adding a subprocess call
wrapper there, but didn't like much the solution, but this was
before the need to use futures.concurrent.
> That said, I've been unable to make this change break in my testing. I
> guess I'm not seeing a lot of impediments to applying the next version
> at this point.
Great! I'll probably be respinning the next (hopefully final) version
by the end of this week, if I don't get sidetracked with other things.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-04 7:33 ` [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
2025-09-09 14:53 ` Jonathan Corbet
@ 2025-09-09 15:21 ` Jonathan Corbet
2025-09-09 16:06 ` Mauro Carvalho Chehab
2025-09-10 10:46 ` Jani Nikula
2 siblings, 1 reply; 67+ messages in thread
From: Jonathan Corbet @ 2025-09-09 15:21 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Andreas Hindborg, Benno Lossin, Boqun Feng,
Danilo Krummrich, Gary Guo, Mauro Carvalho Chehab, Miguel Ojeda,
Trevor Gross, linux-kernel, rust-for-linux
Another nit:
> # sphinxdirs can be a list or a whitespace-separated string
> #
> sphinxdirs_list = []
> for sphinxdir in sphinxdirs:
> if isinstance(sphinxdir, list):
> sphinxdirs_list += sphinxdir
> else:
> for name in sphinxdir.split(" "):
> sphinxdirs_list.append(name)
That inner loop just seems like a complicated way of saying:
sphinxdirs_list += sphinxdir.split()
?
Thanks,
jon
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-09 15:21 ` Jonathan Corbet
@ 2025-09-09 16:06 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-09 16:06 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
On Tue, Sep 09, 2025 at 09:21:35AM -0600, Jonathan Corbet wrote:
> Another nit:
>
> > # sphinxdirs can be a list or a whitespace-separated string
> > #
> > sphinxdirs_list = []
> > for sphinxdir in sphinxdirs:
> > if isinstance(sphinxdir, list):
> > sphinxdirs_list += sphinxdir
> > else:
> > for name in sphinxdir.split(" "):
> > sphinxdirs_list.append(name)
>
> That inner loop just seems like a complicated way of saying:
>
> sphinxdirs_list += sphinxdir.split()
Yeah, it sounds so ;-)
At the development code version, I had some prints there to be sure
all cases were picked, so I ended coding it as a loop. I forgot to
return it to the much nicer "+=" syntax after finishing debugging it.
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-04 7:33 ` [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
2025-09-09 14:53 ` Jonathan Corbet
2025-09-09 15:21 ` Jonathan Corbet
@ 2025-09-10 10:46 ` Jani Nikula
2025-09-10 12:59 ` Mauro Carvalho Chehab
2 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-10 10:46 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Andreas Hindborg, Benno Lossin, Boqun Feng,
Danilo Krummrich, Gary Guo, Mauro Carvalho Chehab, Miguel Ojeda,
Trevor Gross, linux-kernel, rust-for-linux
On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> There are too much magic inside docs Makefile to properly run
> sphinx-build. Create an ancillary script that contains all
> kernel-related sphinx-build call logic currently at Makefile.
>
> Such script is designed to work both as an standalone command
> and as part of a Makefile. As such, it properly handles POSIX
> jobserver used by GNU make.
>
> On a side note, there was a line number increase due to the
> conversion:
>
> Documentation/Makefile | 131 +++----------
> tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++
> 2 files changed, 323 insertions(+), 101 deletions(-)
>
> This is because some things are more verbosed on Python and because
> it requires reading env vars from Makefile. Besides it, this script
> has some extra features that don't exist at the Makefile:
>
> - It can be called directly from command line;
> - It properly return PDF build errors.
>
> When running the script alone, it will only take handle sphinx-build
> targets. On other words, it won't runn make rustdoc after building
> htmlfiles, nor it will run the extra check scripts.
I've always strongly believed we should aim to make it possible to build
the documentation by running sphinx-build directly on the
command-line. Not that it would be the common way to run it, but to not
accumulate things in the Makefile that need to happen before or
after. To promote handling the documentation build in Sphinx. To be able
to debug issues and try new Sphinx versions without all the hacks.
This patch moves a bunch of that logic into a Python wrapper, and I feel
like it complicates matters. You can no longer rely on 'make V=1' to get
the build commands, for instance.
Newer Sphinx versions have the -M option for "make mode". The Makefiles
produced by sphinx-quickstart only have one build target:
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
That's all.
The proposed wrapper duplicates loads of code that's supposed to be
handled by sphinx-build directly. Including the target/builder names.
Seems to me the goal should be to figure out *generic* wrappers for
handling parallelism, not Sphinx aware/specific.
BR,
Jani.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> Documentation/Makefile | 131 ++++----------
> tools/docs/sphinx-build-wrapper | 293 ++++++++++++++++++++++++++++++++
> 2 files changed, 323 insertions(+), 101 deletions(-)
> create mode 100755 tools/docs/sphinx-build-wrapper
>
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index deb2029228ed..4736f02b6c9e 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -23,21 +23,22 @@ SPHINXOPTS =
> SPHINXDIRS = .
> DOCS_THEME =
> DOCS_CSS =
> -_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
> SPHINX_CONF = conf.py
> PAPER =
> BUILDDIR = $(obj)/output
> PDFLATEX = xelatex
> LATEXOPTS = -interaction=batchmode -no-shell-escape
>
> +PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
> +
> +# Wrapper for sphinx-build
> +
> +BUILD_WRAPPER = $(srctree)/tools/docs/sphinx-build-wrapper
> +
> # For denylisting "variable font" files
> # Can be overridden by setting as an env variable
> FONTS_CONF_DENY_VF ?= $(HOME)/deny-vf
>
> -ifeq ($(findstring 1, $(KBUILD_VERBOSE)),)
> -SPHINXOPTS += "-q"
> -endif
> -
> # User-friendly check for sphinx-build
> HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)
>
> @@ -51,63 +52,31 @@ ifeq ($(HAVE_SPHINX),0)
>
> else # HAVE_SPHINX
>
> -# User-friendly check for pdflatex and latexmk
> -HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
> -HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
> +# Common documentation targets
> +infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
> + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> + --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> + --builddir="$(BUILDDIR)" \
> + --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
>
> -ifeq ($(HAVE_LATEXMK),1)
> - PDFLATEX := latexmk -$(PDFLATEX)
> -endif #HAVE_LATEXMK
> -
> -# Internal variables.
> -PAPEROPT_a4 = -D latex_elements.papersize=a4paper
> -PAPEROPT_letter = -D latex_elements.papersize=letterpaper
> -ALLSPHINXOPTS = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
> -ALLSPHINXOPTS += $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
> -ifneq ($(wildcard $(srctree)/.config),)
> -ifeq ($(CONFIG_RUST),y)
> - # Let Sphinx know we will include rustdoc
> - ALLSPHINXOPTS += -t rustdoc
> -endif
> +# Special handling for pdfdocs
> +ifeq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0)
> +pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
> +else
> +pdfdocs:
> + $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> + @echo " SKIP Sphinx $@ target."
> endif
> -# the i18n builder cannot share the environment and doctrees with the others
> -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
> -
> -# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
> -loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
> -
> -# $2 sphinx builder e.g. "html"
> -# $3 name of the build subfolder / e.g. "userspace-api/media", used as:
> -# * dest folder relative to $(BUILDDIR) and
> -# * cache folder relative to $(BUILDDIR)/.doctrees
> -# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man
> -# $5 reST source folder relative to $(src),
> -# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentation/userspace-api/media
> -
> -PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
> -
> -quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
> - cmd_sphinx = \
> - PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
> - BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \
> - $(PYTHON3) $(srctree)/scripts/jobserver-exec \
> - $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
> - $(SPHINXBUILD) \
> - -b $2 \
> - -c $(abspath $(src)) \
> - -d $(abspath $(BUILDDIR)/.doctrees/$3) \
> - -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
> - $(ALLSPHINXOPTS) \
> - $(abspath $(src)/$5) \
> - $(abspath $(BUILDDIR)/$3/$4) && \
> - if [ "x$(DOCS_CSS)" != "x" ]; then \
> - cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(DOCS_CSS)) $(BUILDDIR)/$3/_static/; \
> - fi
>
> +# HTML main logic is identical to other targets. However, if rust is enabled,
> +# an extra step at the end is required to generate rustdoc.
> htmldocs:
> - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
> -
> + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> + --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> + --builddir="$(BUILDDIR)" \
> + --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
> # If Rust support is available and .config exists, add rustdoc generated contents.
> # If there are any, the errors from this make rustdoc will be displayed but
> # won't stop the execution of htmldocs
> @@ -118,49 +87,6 @@ ifeq ($(CONFIG_RUST),y)
> endif
> endif
>
> -texinfodocs:
> - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
> -
> -# Note: the 'info' Make target is generated by sphinx itself when
> -# running the texinfodocs target define above.
> -infodocs: texinfodocs
> - $(MAKE) -C $(BUILDDIR)/texinfo info
> -
> -linkcheckdocs:
> - @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
> -
> -latexdocs:
> - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
> -
> -ifeq ($(HAVE_PDFLATEX),0)
> -
> -pdfdocs:
> - $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> - @echo " SKIP Sphinx $@ target."
> -
> -else # HAVE_PDFLATEX
> -
> -pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
> -pdfdocs: latexdocs
> - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> - $(foreach var,$(SPHINXDIRS), \
> - $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
> - mkdir -p $(BUILDDIR)/$(var)/pdf; \
> - mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
> - )
> -
> -endif # HAVE_PDFLATEX
> -
> -epubdocs:
> - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
> -
> -xmldocs:
> - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
> -
> endif # HAVE_SPHINX
>
> # The following targets are independent of HAVE_SPHINX, and the rules should
> @@ -172,6 +98,9 @@ refcheckdocs:
> cleandocs:
> $(Q)rm -rf $(BUILDDIR)
>
> +# Used only on help
> +_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
> +
> dochelp:
> @echo ' Linux kernel internal documentation in different formats from ReST:'
> @echo ' htmldocs - HTML'
> diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
> new file mode 100755
> index 000000000000..3256418d8dc5
> --- /dev/null
> +++ b/tools/docs/sphinx-build-wrapper
> @@ -0,0 +1,293 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: GPL-2.0
> +import argparse
> +import os
> +import shlex
> +import shutil
> +import subprocess
> +import sys
> +from lib.python_version import PythonVersion
> +
> +LIB_DIR = "../../scripts/lib"
> +SRC_DIR = os.path.dirname(os.path.realpath(__file__))
> +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
> +
> +from jobserver import JobserverExec
> +
> +MIN_PYTHON_VERSION = PythonVersion("3.7").version
> +PAPER = ["", "a4", "letter"]
> +TARGETS = {
> + "cleandocs": { "builder": "clean" },
> + "linkcheckdocs": { "builder": "linkcheck" },
> + "htmldocs": { "builder": "html" },
> + "epubdocs": { "builder": "epub", "out_dir": "epub" },
> + "texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" },
> + "infodocs": { "builder": "texinfo", "out_dir": "texinfo" },
> + "latexdocs": { "builder": "latex", "out_dir": "latex" },
> + "pdfdocs": { "builder": "latex", "out_dir": "latex" },
> + "xmldocs": { "builder": "xml", "out_dir": "xml" },
> +}
> +
> +class SphinxBuilder:
> + def is_rust_enabled(self):
> + config_path = os.path.join(self.srctree, ".config")
> + if os.path.isfile(config_path):
> + with open(config_path, "r", encoding="utf-8") as f:
> + return "CONFIG_RUST=y" in f.read()
> + return False
> +
> + def get_path(self, path, use_cwd=False, abs_path=False):
> + path = os.path.expanduser(path)
> + if not path.startswith("/"):
> + if use_cwd:
> + base = os.getcwd()
> + else:
> + base = self.srctree
> + path = os.path.join(base, path)
> + if abs_path:
> + return os.path.abspath(path)
> + return path
> +
> + def __init__(self, builddir, verbose=False, n_jobs=None):
> + self.verbose = None
> + self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
> + self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
> + self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
> + self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
> + if not verbose:
> + verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
> + if verbose is not None:
> + self.verbose = verbose
> + parser = argparse.ArgumentParser()
> + parser.add_argument('-j', '--jobs', type=int)
> + parser.add_argument('-q', '--quiet', type=int)
> + sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", ""))
> + sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
> + if sphinx_args.quiet is True:
> + self.verbose = False
> + if sphinx_args.jobs:
> + self.n_jobs = sphinx_args.jobs
> + self.n_jobs = n_jobs
> + self.srctree = os.environ.get("srctree")
> + if not self.srctree:
> + self.srctree = "."
> + os.environ["srctree"] = self.srctree
> + self.sphinxbuild = os.environ.get("SPHINXBUILD", "sphinx-build")
> + self.kerneldoc = self.get_path(os.environ.get("KERNELDOC",
> + "scripts/kernel-doc.py"))
> + self.builddir = self.get_path(builddir, use_cwd=True, abs_path=True)
> +
> + self.config_rust = self.is_rust_enabled()
> +
> + self.pdflatex_cmd = shutil.which(self.pdflatex)
> + self.latexmk_cmd = shutil.which("latexmk")
> +
> + self.env = os.environ.copy()
> +
> + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
> + with JobserverExec() as jobserver:
> + if jobserver.claim:
> + n_jobs = str(jobserver.claim)
> + else:
> + n_jobs = "auto" # Supported since Sphinx 1.7
> + cmd = []
> + cmd.append(sys.executable)
> + cmd.append(sphinx_build)
> + if self.n_jobs:
> + n_jobs = str(self.n_jobs)
> +
> + if n_jobs:
> + cmd += [f"-j{n_jobs}"]
> +
> + if not self.verbose:
> + cmd.append("-q")
> + cmd += self.sphinxopts
> + cmd += build_args
> + if self.verbose:
> + print(" ".join(cmd))
> + return subprocess.call(cmd, *args, **pwargs)
> +
> + def handle_html(self, css, output_dir):
> + if not css:
> + return
> + css = os.path.expanduser(css)
> + if not css.startswith("/"):
> + css = os.path.join(self.srctree, css)
> + static_dir = os.path.join(output_dir, "_static")
> + os.makedirs(static_dir, exist_ok=True)
> + try:
> + shutil.copy2(css, static_dir)
> + except (OSError, IOError) as e:
> + print(f"Warning: Failed to copy CSS: {e}", file=sys.stderr)
> +
> + def handle_pdf(self, output_dirs):
> + builds = {}
> + max_len = 0
> + for from_dir in output_dirs:
> + pdf_dir = os.path.join(from_dir, "../pdf")
> + os.makedirs(pdf_dir, exist_ok=True)
> + if self.latexmk_cmd:
> + latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
> + else:
> + latex_cmd = [self.pdflatex]
> + latex_cmd.extend(shlex.split(self.latexopts))
> + tex_suffix = ".tex"
> + has_tex = False
> + build_failed = False
> + with os.scandir(from_dir) as it:
> + for entry in it:
> + if not entry.name.endswith(tex_suffix):
> + continue
> + name = entry.name[:-len(tex_suffix)]
> + has_tex = True
> + try:
> + subprocess.run(latex_cmd + [entry.path],
> + cwd=from_dir, check=True)
> + except subprocess.CalledProcessError:
> + pass
> + pdf_name = name + ".pdf"
> + pdf_from = os.path.join(from_dir, pdf_name)
> + pdf_to = os.path.join(pdf_dir, pdf_name)
> + if os.path.exists(pdf_from):
> + os.rename(pdf_from, pdf_to)
> + builds[name] = os.path.relpath(pdf_to, self.builddir)
> + else:
> + builds[name] = "FAILED"
> + build_failed = True
> + name = entry.name.removesuffix(".tex")
> + max_len = max(max_len, len(name))
> +
> + if not has_tex:
> + name = os.path.basename(from_dir)
> + max_len = max(max_len, len(name))
> + builds[name] = "FAILED (no .tex)"
> + build_failed = True
> + msg = "Summary"
> + msg += "\n" + "=" * len(msg)
> + print()
> + print(msg)
> + for pdf_name, pdf_file in builds.items():
> + print(f"{pdf_name:<{max_len}}: {pdf_file}")
> + print()
> + if build_failed:
> + sys.exit("PDF build failed: not all PDF files were created.")
> + else:
> + print("All PDF files were built.")
> +
> + def handle_info(self, output_dirs):
> + for output_dir in output_dirs:
> + try:
> + subprocess.run(["make", "info"], cwd=output_dir, check=True)
> + except subprocess.CalledProcessError as e:
> + sys.exit(f"Error generating info docs: {e}")
> +
> + def cleandocs(self, builder):
> + shutil.rmtree(self.builddir, ignore_errors=True)
> +
> + def build(self, target, sphinxdirs=None, conf="conf.py",
> + theme=None, css=None, paper=None):
> + builder = TARGETS[target]["builder"]
> + out_dir = TARGETS[target].get("out_dir", "")
> + if target == "cleandocs":
> + self.cleandocs(builder)
> + return
> + if theme:
> + os.environ["DOCS_THEME"] = theme
> + sphinxbuild = shutil.which(self.sphinxbuild, path=self.env["PATH"])
> + if not sphinxbuild:
> + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n")
> + if builder == "latex":
> + if not self.pdflatex_cmd and not self.latexmk_cmd:
> + sys.exit("Error: pdflatex or latexmk required for PDF generation")
> + docs_dir = os.path.abspath(os.path.join(self.srctree, "Documentation"))
> + kerneldoc = self.kerneldoc
> + if kerneldoc.startswith(self.srctree):
> + kerneldoc = os.path.relpath(kerneldoc, self.srctree)
> + args = [ "-b", builder, "-c", docs_dir ]
> + if builder == "latex":
> + if not paper:
> + paper = PAPER[1]
> + args.extend(["-D", f"latex_elements.papersize={paper}paper"])
> + if self.config_rust:
> + args.extend(["-t", "rustdoc"])
> + if conf:
> + self.env["SPHINX_CONF"] = self.get_path(conf, abs_path=True)
> + if not sphinxdirs:
> + sphinxdirs = os.environ.get("SPHINXDIRS", ".")
> + sphinxdirs_list = []
> + for sphinxdir in sphinxdirs:
> + if isinstance(sphinxdir, list):
> + sphinxdirs_list += sphinxdir
> + else:
> + for name in sphinxdir.split(" "):
> + sphinxdirs_list.append(name)
> + output_dirs = []
> + for sphinxdir in sphinxdirs_list:
> + src_dir = os.path.join(docs_dir, sphinxdir)
> + doctree_dir = os.path.join(self.builddir, ".doctrees")
> + output_dir = os.path.join(self.builddir, sphinxdir, out_dir)
> + src_dir = os.path.normpath(src_dir)
> + doctree_dir = os.path.normpath(doctree_dir)
> + output_dir = os.path.normpath(output_dir)
> + os.makedirs(doctree_dir, exist_ok=True)
> + os.makedirs(output_dir, exist_ok=True)
> + output_dirs.append(output_dir)
> + build_args = args + [
> + "-d", doctree_dir,
> + "-D", f"kerneldoc_bin={kerneldoc}",
> + "-D", f"version={self.kernelversion}",
> + "-D", f"release={self.kernelrelease}",
> + "-D", f"kerneldoc_srctree={self.srctree}",
> + src_dir,
> + output_dir,
> + ]
> + try:
> + self.run_sphinx(sphinxbuild, build_args, env=self.env)
> + except (OSError, ValueError, subprocess.SubprocessError) as e:
> + sys.exit(f"Build failed: {repr(e)}")
> + if target in ["htmldocs", "epubdocs"]:
> + self.handle_html(css, output_dir)
> + if target == "pdfdocs":
> + self.handle_pdf(output_dirs)
> + elif target == "infodocs":
> + self.handle_info(output_dirs)
> +
> +def jobs_type(value):
> + if value is None:
> + return None
> + if value.lower() == 'auto':
> + return value.lower()
> + try:
> + if int(value) >= 1:
> + return value
> + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}")
> + except ValueError:
> + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}")
> +
> +def main():
> + parser = argparse.ArgumentParser(description="Kernel documentation builder")
> + parser.add_argument("target", choices=list(TARGETS.keys()),
> + help="Documentation target to build")
> + parser.add_argument("--sphinxdirs", nargs="+",
> + help="Specific directories to build")
> + parser.add_argument("--conf", default="conf.py",
> + help="Sphinx configuration file")
> + parser.add_argument("--builddir", default="output",
> + help="Sphinx configuration file")
> + parser.add_argument("--theme", help="Sphinx theme to use")
> + parser.add_argument("--css", help="Custom CSS file for HTML/EPUB")
> + parser.add_argument("--paper", choices=PAPER, default=PAPER[0],
> + help="Paper size for LaTeX/PDF output")
> + parser.add_argument("-v", "--verbose", action='store_true',
> + help="place build in verbose mode")
> + parser.add_argument('-j', '--jobs', type=jobs_type,
> + help="Sets number of jobs to use with sphinx-build")
> + args = parser.parse_args()
> + PythonVersion.check_python(MIN_PYTHON_VERSION)
> + builder = SphinxBuilder(builddir=args.builddir,
> + verbose=args.verbose, n_jobs=args.jobs)
> + builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
> + theme=args.theme, css=args.css, paper=args.paper)
> +
> +if __name__ == "__main__":
> + main()
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-10 10:46 ` Jani Nikula
@ 2025-09-10 12:59 ` Mauro Carvalho Chehab
2025-09-10 13:33 ` Mauro Carvalho Chehab
2025-09-11 10:23 ` Jani Nikula
0 siblings, 2 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-10 12:59 UTC (permalink / raw)
To: Jani Nikula
Cc: Jonathan Corbet, Linux Doc Mailing List, Björn Roy Baron,
Alex Gaynor, Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross,
linux-kernel, rust-for-linux
Em Wed, 10 Sep 2025 13:46:17 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > There are too much magic inside docs Makefile to properly run
> > sphinx-build. Create an ancillary script that contains all
> > kernel-related sphinx-build call logic currently at Makefile.
> >
> > Such script is designed to work both as an standalone command
> > and as part of a Makefile. As such, it properly handles POSIX
> > jobserver used by GNU make.
> >
> > On a side note, there was a line number increase due to the
> > conversion:
> >
> > Documentation/Makefile | 131 +++----------
> > tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++
> > 2 files changed, 323 insertions(+), 101 deletions(-)
> >
> > This is because some things are more verbosed on Python and because
> > it requires reading env vars from Makefile. Besides it, this script
> > has some extra features that don't exist at the Makefile:
> >
> > - It can be called directly from command line;
> > - It properly return PDF build errors.
> >
> > When running the script alone, it will only take handle sphinx-build
> > targets. On other words, it won't runn make rustdoc after building
> > htmlfiles, nor it will run the extra check scripts.
>
> I've always strongly believed we should aim to make it possible to build
> the documentation by running sphinx-build directly on the
> command-line. Not that it would be the common way to run it, but to not
> accumulate things in the Makefile that need to happen before or
> after. To promote handling the documentation build in Sphinx. To be able
> to debug issues and try new Sphinx versions without all the hacks.
That would be the better, but, unfortunately, this is not possible, for
several reasons:
1. SPHINXDIRS. It needs a lot of magic to work, both before running
sphinx-build and after (inside conf.py);
2. Several extensions require kernel-specific environment variables to
work. Calling sphinx-build directly breaks them;
3. Sphinx itself doesn't build several targets alone. Instead, they create
a Makefile, and an extra step is needed to finish the build. That's
the case for pdf and texinfo, for instance;
4. Man pages generation. Sphinx support to generate it is very poor;
5. Rust integration adds more complexity to the table;
I'm not seeing sphinx-build supporting the above needs anytime soon,
and, even if we push our needs to Sphinx and it gets accepted there,
we'll still need to wait for quite a while until LTS distros merge
them.
> This patch moves a bunch of that logic into a Python wrapper, and I feel
> like it complicates matters. You can no longer rely on 'make V=1' to get
> the build commands, for instance.
Quite the opposite. if you try using "make V=1", it won't show the
command line used to call sphinx-build anymore.
This series restore it.
See, if you build with this series with V=1, you will see exactly
what commands are used on the build:
$ make V=1 htmldocs
...
python3 ./tools/docs/sphinx-build-wrapper htmldocs \
--sphinxdirs="." --conf="conf.py" \
--builddir="Documentation/output" \
--theme= --css= --paper=
python3 /new_devel/docs/sphinx_latest/bin/sphinx-build -j25 -b html -c /new_devel/docs/Documentation -d /new_devel/docs/Documentation/output/.doctrees -D kerneldoc_bin=scripts/kernel-doc.py -D version=6.17.0-rc1 -D release=6.17.0-rc1+ -D kerneldoc_srctree=. /new_devel/docs/Documentation /new_devel/docs/Documentation/output
...
> Newer Sphinx versions have the -M option for "make mode". The Makefiles
> produced by sphinx-quickstart only have one build target:
>
> # Catch-all target: route all unknown targets to Sphinx using the new
> # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
I didn't know about this, but from [1] it sounds it covers just two
targets: "latexpdf" and "info".
The most complex scenario is still not covered: SPHINXDIRS.
[1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html
> %: Makefile
> @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
>
> That's all.
Try doing such change on your makefile. it will break:
- SPHINXDIRS;
- V=1;
- rustdoc
and will still be dependent on variables that are passed via
env from Kernel makefile. So, stil you can't run from command
line. Also, if you call sphinx-build from command line:
$ sphinx-build -j25 -b html Documentation Documentation/output
...
File "<frozen os>", line 717, in __getitem__
KeyError: 'srctree'
It won't work, as several parameters that are required by conf.py and by
Sphinx extensions would be missing (the most important one is srctree, but
there are others in the line too).
> The proposed wrapper duplicates loads of code that's supposed to be
> handled by sphinx-build directly.
Once we get the wrapper, we can work to simplify it, but still I
can't see how to get rid of it.
> Including the target/builder names.
True, but this was a design decision taken lots of years ago: instead
of:
make html
we're using:
make htmldocs
This series doesn't change that: either makefile or the script need
to tho the namespace conversion.
> Seems to me the goal should be to figure out *generic* wrappers for
> handling parallelism, not Sphinx aware/specific.
>
>
> BR,
> Jani.
>
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > Documentation/Makefile | 131 ++++----------
> > tools/docs/sphinx-build-wrapper | 293 ++++++++++++++++++++++++++++++++
> > 2 files changed, 323 insertions(+), 101 deletions(-)
> > create mode 100755 tools/docs/sphinx-build-wrapper
> >
> > diff --git a/Documentation/Makefile b/Documentation/Makefile
> > index deb2029228ed..4736f02b6c9e 100644
> > --- a/Documentation/Makefile
> > +++ b/Documentation/Makefile
> > @@ -23,21 +23,22 @@ SPHINXOPTS =
> > SPHINXDIRS = .
> > DOCS_THEME =
> > DOCS_CSS =
> > -_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
> > SPHINX_CONF = conf.py
> > PAPER =
> > BUILDDIR = $(obj)/output
> > PDFLATEX = xelatex
> > LATEXOPTS = -interaction=batchmode -no-shell-escape
> >
> > +PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
> > +
> > +# Wrapper for sphinx-build
> > +
> > +BUILD_WRAPPER = $(srctree)/tools/docs/sphinx-build-wrapper
> > +
> > # For denylisting "variable font" files
> > # Can be overridden by setting as an env variable
> > FONTS_CONF_DENY_VF ?= $(HOME)/deny-vf
> >
> > -ifeq ($(findstring 1, $(KBUILD_VERBOSE)),)
> > -SPHINXOPTS += "-q"
> > -endif
> > -
> > # User-friendly check for sphinx-build
> > HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)
> >
> > @@ -51,63 +52,31 @@ ifeq ($(HAVE_SPHINX),0)
> >
> > else # HAVE_SPHINX
> >
> > -# User-friendly check for pdflatex and latexmk
> > -HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
> > -HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
> > +# Common documentation targets
> > +infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
> > + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> > + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> > + --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> > + --builddir="$(BUILDDIR)" \
> > + --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
> >
> > -ifeq ($(HAVE_LATEXMK),1)
> > - PDFLATEX := latexmk -$(PDFLATEX)
> > -endif #HAVE_LATEXMK
> > -
> > -# Internal variables.
> > -PAPEROPT_a4 = -D latex_elements.papersize=a4paper
> > -PAPEROPT_letter = -D latex_elements.papersize=letterpaper
> > -ALLSPHINXOPTS = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
> > -ALLSPHINXOPTS += $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
> > -ifneq ($(wildcard $(srctree)/.config),)
> > -ifeq ($(CONFIG_RUST),y)
> > - # Let Sphinx know we will include rustdoc
> > - ALLSPHINXOPTS += -t rustdoc
> > -endif
> > +# Special handling for pdfdocs
> > +ifeq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0)
> > +pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
> > +else
> > +pdfdocs:
> > + $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> > + @echo " SKIP Sphinx $@ target."
> > endif
> > -# the i18n builder cannot share the environment and doctrees with the others
> > -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
> > -
> > -# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
> > -loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
> > -
> > -# $2 sphinx builder e.g. "html"
> > -# $3 name of the build subfolder / e.g. "userspace-api/media", used as:
> > -# * dest folder relative to $(BUILDDIR) and
> > -# * cache folder relative to $(BUILDDIR)/.doctrees
> > -# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man
> > -# $5 reST source folder relative to $(src),
> > -# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentation/userspace-api/media
> > -
> > -PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
> > -
> > -quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
> > - cmd_sphinx = \
> > - PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
> > - BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \
> > - $(PYTHON3) $(srctree)/scripts/jobserver-exec \
> > - $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
> > - $(SPHINXBUILD) \
> > - -b $2 \
> > - -c $(abspath $(src)) \
> > - -d $(abspath $(BUILDDIR)/.doctrees/$3) \
> > - -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
> > - $(ALLSPHINXOPTS) \
> > - $(abspath $(src)/$5) \
> > - $(abspath $(BUILDDIR)/$3/$4) && \
> > - if [ "x$(DOCS_CSS)" != "x" ]; then \
> > - cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(DOCS_CSS)) $(BUILDDIR)/$3/_static/; \
> > - fi
> >
> > +# HTML main logic is identical to other targets. However, if rust is enabled,
> > +# an extra step at the end is required to generate rustdoc.
> > htmldocs:
> > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
> > -
> > + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> > + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> > + --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> > + --builddir="$(BUILDDIR)" \
> > + --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
> > # If Rust support is available and .config exists, add rustdoc generated contents.
> > # If there are any, the errors from this make rustdoc will be displayed but
> > # won't stop the execution of htmldocs
> > @@ -118,49 +87,6 @@ ifeq ($(CONFIG_RUST),y)
> > endif
> > endif
> >
> > -texinfodocs:
> > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
> > -
> > -# Note: the 'info' Make target is generated by sphinx itself when
> > -# running the texinfodocs target define above.
> > -infodocs: texinfodocs
> > - $(MAKE) -C $(BUILDDIR)/texinfo info
> > -
> > -linkcheckdocs:
> > - @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
> > -
> > -latexdocs:
> > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
> > -
> > -ifeq ($(HAVE_PDFLATEX),0)
> > -
> > -pdfdocs:
> > - $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> > - @echo " SKIP Sphinx $@ target."
> > -
> > -else # HAVE_PDFLATEX
> > -
> > -pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
> > -pdfdocs: latexdocs
> > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > - $(foreach var,$(SPHINXDIRS), \
> > - $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
> > - mkdir -p $(BUILDDIR)/$(var)/pdf; \
> > - mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
> > - )
> > -
> > -endif # HAVE_PDFLATEX
> > -
> > -epubdocs:
> > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
> > -
> > -xmldocs:
> > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
> > -
> > endif # HAVE_SPHINX
> >
> > # The following targets are independent of HAVE_SPHINX, and the rules should
> > @@ -172,6 +98,9 @@ refcheckdocs:
> > cleandocs:
> > $(Q)rm -rf $(BUILDDIR)
> >
> > +# Used only on help
> > +_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
> > +
> > dochelp:
> > @echo ' Linux kernel internal documentation in different formats from ReST:'
> > @echo ' htmldocs - HTML'
> > diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
> > new file mode 100755
> > index 000000000000..3256418d8dc5
> > --- /dev/null
> > +++ b/tools/docs/sphinx-build-wrapper
> > @@ -0,0 +1,293 @@
> > +#!/usr/bin/env python3
> > +# SPDX-License-Identifier: GPL-2.0
> > +import argparse
> > +import os
> > +import shlex
> > +import shutil
> > +import subprocess
> > +import sys
> > +from lib.python_version import PythonVersion
> > +
> > +LIB_DIR = "../../scripts/lib"
> > +SRC_DIR = os.path.dirname(os.path.realpath(__file__))
> > +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
> > +
> > +from jobserver import JobserverExec
> > +
> > +MIN_PYTHON_VERSION = PythonVersion("3.7").version
> > +PAPER = ["", "a4", "letter"]
> > +TARGETS = {
> > + "cleandocs": { "builder": "clean" },
> > + "linkcheckdocs": { "builder": "linkcheck" },
> > + "htmldocs": { "builder": "html" },
> > + "epubdocs": { "builder": "epub", "out_dir": "epub" },
> > + "texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" },
> > + "infodocs": { "builder": "texinfo", "out_dir": "texinfo" },
> > + "latexdocs": { "builder": "latex", "out_dir": "latex" },
> > + "pdfdocs": { "builder": "latex", "out_dir": "latex" },
> > + "xmldocs": { "builder": "xml", "out_dir": "xml" },
> > +}
> > +
> > +class SphinxBuilder:
> > + def is_rust_enabled(self):
> > + config_path = os.path.join(self.srctree, ".config")
> > + if os.path.isfile(config_path):
> > + with open(config_path, "r", encoding="utf-8") as f:
> > + return "CONFIG_RUST=y" in f.read()
> > + return False
> > +
> > + def get_path(self, path, use_cwd=False, abs_path=False):
> > + path = os.path.expanduser(path)
> > + if not path.startswith("/"):
> > + if use_cwd:
> > + base = os.getcwd()
> > + else:
> > + base = self.srctree
> > + path = os.path.join(base, path)
> > + if abs_path:
> > + return os.path.abspath(path)
> > + return path
> > +
> > + def __init__(self, builddir, verbose=False, n_jobs=None):
> > + self.verbose = None
> > + self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
> > + self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
> > + self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
> > + self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
> > + if not verbose:
> > + verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
> > + if verbose is not None:
> > + self.verbose = verbose
> > + parser = argparse.ArgumentParser()
> > + parser.add_argument('-j', '--jobs', type=int)
> > + parser.add_argument('-q', '--quiet', type=int)
> > + sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", ""))
> > + sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
> > + if sphinx_args.quiet is True:
> > + self.verbose = False
> > + if sphinx_args.jobs:
> > + self.n_jobs = sphinx_args.jobs
> > + self.n_jobs = n_jobs
> > + self.srctree = os.environ.get("srctree")
> > + if not self.srctree:
> > + self.srctree = "."
> > + os.environ["srctree"] = self.srctree
> > + self.sphinxbuild = os.environ.get("SPHINXBUILD", "sphinx-build")
> > + self.kerneldoc = self.get_path(os.environ.get("KERNELDOC",
> > + "scripts/kernel-doc.py"))
> > + self.builddir = self.get_path(builddir, use_cwd=True, abs_path=True)
> > +
> > + self.config_rust = self.is_rust_enabled()
> > +
> > + self.pdflatex_cmd = shutil.which(self.pdflatex)
> > + self.latexmk_cmd = shutil.which("latexmk")
> > +
> > + self.env = os.environ.copy()
> > +
> > + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
> > + with JobserverExec() as jobserver:
> > + if jobserver.claim:
> > + n_jobs = str(jobserver.claim)
> > + else:
> > + n_jobs = "auto" # Supported since Sphinx 1.7
> > + cmd = []
> > + cmd.append(sys.executable)
> > + cmd.append(sphinx_build)
> > + if self.n_jobs:
> > + n_jobs = str(self.n_jobs)
> > +
> > + if n_jobs:
> > + cmd += [f"-j{n_jobs}"]
> > +
> > + if not self.verbose:
> > + cmd.append("-q")
> > + cmd += self.sphinxopts
> > + cmd += build_args
> > + if self.verbose:
> > + print(" ".join(cmd))
> > + return subprocess.call(cmd, *args, **pwargs)
> > +
> > + def handle_html(self, css, output_dir):
> > + if not css:
> > + return
> > + css = os.path.expanduser(css)
> > + if not css.startswith("/"):
> > + css = os.path.join(self.srctree, css)
> > + static_dir = os.path.join(output_dir, "_static")
> > + os.makedirs(static_dir, exist_ok=True)
> > + try:
> > + shutil.copy2(css, static_dir)
> > + except (OSError, IOError) as e:
> > + print(f"Warning: Failed to copy CSS: {e}", file=sys.stderr)
> > +
> > + def handle_pdf(self, output_dirs):
> > + builds = {}
> > + max_len = 0
> > + for from_dir in output_dirs:
> > + pdf_dir = os.path.join(from_dir, "../pdf")
> > + os.makedirs(pdf_dir, exist_ok=True)
> > + if self.latexmk_cmd:
> > + latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
> > + else:
> > + latex_cmd = [self.pdflatex]
> > + latex_cmd.extend(shlex.split(self.latexopts))
> > + tex_suffix = ".tex"
> > + has_tex = False
> > + build_failed = False
> > + with os.scandir(from_dir) as it:
> > + for entry in it:
> > + if not entry.name.endswith(tex_suffix):
> > + continue
> > + name = entry.name[:-len(tex_suffix)]
> > + has_tex = True
> > + try:
> > + subprocess.run(latex_cmd + [entry.path],
> > + cwd=from_dir, check=True)
> > + except subprocess.CalledProcessError:
> > + pass
> > + pdf_name = name + ".pdf"
> > + pdf_from = os.path.join(from_dir, pdf_name)
> > + pdf_to = os.path.join(pdf_dir, pdf_name)
> > + if os.path.exists(pdf_from):
> > + os.rename(pdf_from, pdf_to)
> > + builds[name] = os.path.relpath(pdf_to, self.builddir)
> > + else:
> > + builds[name] = "FAILED"
> > + build_failed = True
> > + name = entry.name.removesuffix(".tex")
> > + max_len = max(max_len, len(name))
> > +
> > + if not has_tex:
> > + name = os.path.basename(from_dir)
> > + max_len = max(max_len, len(name))
> > + builds[name] = "FAILED (no .tex)"
> > + build_failed = True
> > + msg = "Summary"
> > + msg += "\n" + "=" * len(msg)
> > + print()
> > + print(msg)
> > + for pdf_name, pdf_file in builds.items():
> > + print(f"{pdf_name:<{max_len}}: {pdf_file}")
> > + print()
> > + if build_failed:
> > + sys.exit("PDF build failed: not all PDF files were created.")
> > + else:
> > + print("All PDF files were built.")
> > +
> > + def handle_info(self, output_dirs):
> > + for output_dir in output_dirs:
> > + try:
> > + subprocess.run(["make", "info"], cwd=output_dir, check=True)
> > + except subprocess.CalledProcessError as e:
> > + sys.exit(f"Error generating info docs: {e}")
> > +
> > + def cleandocs(self, builder):
> > + shutil.rmtree(self.builddir, ignore_errors=True)
> > +
> > + def build(self, target, sphinxdirs=None, conf="conf.py",
> > + theme=None, css=None, paper=None):
> > + builder = TARGETS[target]["builder"]
> > + out_dir = TARGETS[target].get("out_dir", "")
> > + if target == "cleandocs":
> > + self.cleandocs(builder)
> > + return
> > + if theme:
> > + os.environ["DOCS_THEME"] = theme
> > + sphinxbuild = shutil.which(self.sphinxbuild, path=self.env["PATH"])
> > + if not sphinxbuild:
> > + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n")
> > + if builder == "latex":
> > + if not self.pdflatex_cmd and not self.latexmk_cmd:
> > + sys.exit("Error: pdflatex or latexmk required for PDF generation")
> > + docs_dir = os.path.abspath(os.path.join(self.srctree, "Documentation"))
> > + kerneldoc = self.kerneldoc
> > + if kerneldoc.startswith(self.srctree):
> > + kerneldoc = os.path.relpath(kerneldoc, self.srctree)
> > + args = [ "-b", builder, "-c", docs_dir ]
> > + if builder == "latex":
> > + if not paper:
> > + paper = PAPER[1]
> > + args.extend(["-D", f"latex_elements.papersize={paper}paper"])
> > + if self.config_rust:
> > + args.extend(["-t", "rustdoc"])
> > + if conf:
> > + self.env["SPHINX_CONF"] = self.get_path(conf, abs_path=True)
> > + if not sphinxdirs:
> > + sphinxdirs = os.environ.get("SPHINXDIRS", ".")
> > + sphinxdirs_list = []
> > + for sphinxdir in sphinxdirs:
> > + if isinstance(sphinxdir, list):
> > + sphinxdirs_list += sphinxdir
> > + else:
> > + for name in sphinxdir.split(" "):
> > + sphinxdirs_list.append(name)
> > + output_dirs = []
> > + for sphinxdir in sphinxdirs_list:
> > + src_dir = os.path.join(docs_dir, sphinxdir)
> > + doctree_dir = os.path.join(self.builddir, ".doctrees")
> > + output_dir = os.path.join(self.builddir, sphinxdir, out_dir)
> > + src_dir = os.path.normpath(src_dir)
> > + doctree_dir = os.path.normpath(doctree_dir)
> > + output_dir = os.path.normpath(output_dir)
> > + os.makedirs(doctree_dir, exist_ok=True)
> > + os.makedirs(output_dir, exist_ok=True)
> > + output_dirs.append(output_dir)
> > + build_args = args + [
> > + "-d", doctree_dir,
> > + "-D", f"kerneldoc_bin={kerneldoc}",
> > + "-D", f"version={self.kernelversion}",
> > + "-D", f"release={self.kernelrelease}",
> > + "-D", f"kerneldoc_srctree={self.srctree}",
> > + src_dir,
> > + output_dir,
> > + ]
> > + try:
> > + self.run_sphinx(sphinxbuild, build_args, env=self.env)
> > + except (OSError, ValueError, subprocess.SubprocessError) as e:
> > + sys.exit(f"Build failed: {repr(e)}")
> > + if target in ["htmldocs", "epubdocs"]:
> > + self.handle_html(css, output_dir)
> > + if target == "pdfdocs":
> > + self.handle_pdf(output_dirs)
> > + elif target == "infodocs":
> > + self.handle_info(output_dirs)
> > +
> > +def jobs_type(value):
> > + if value is None:
> > + return None
> > + if value.lower() == 'auto':
> > + return value.lower()
> > + try:
> > + if int(value) >= 1:
> > + return value
> > + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}")
> > + except ValueError:
> > + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}")
> > +
> > +def main():
> > + parser = argparse.ArgumentParser(description="Kernel documentation builder")
> > + parser.add_argument("target", choices=list(TARGETS.keys()),
> > + help="Documentation target to build")
> > + parser.add_argument("--sphinxdirs", nargs="+",
> > + help="Specific directories to build")
> > + parser.add_argument("--conf", default="conf.py",
> > + help="Sphinx configuration file")
> > + parser.add_argument("--builddir", default="output",
> > + help="Sphinx configuration file")
> > + parser.add_argument("--theme", help="Sphinx theme to use")
> > + parser.add_argument("--css", help="Custom CSS file for HTML/EPUB")
> > + parser.add_argument("--paper", choices=PAPER, default=PAPER[0],
> > + help="Paper size for LaTeX/PDF output")
> > + parser.add_argument("-v", "--verbose", action='store_true',
> > + help="place build in verbose mode")
> > + parser.add_argument('-j', '--jobs', type=jobs_type,
> > + help="Sets number of jobs to use with sphinx-build")
> > + args = parser.parse_args()
> > + PythonVersion.check_python(MIN_PYTHON_VERSION)
> > + builder = SphinxBuilder(builddir=args.builddir,
> > + verbose=args.verbose, n_jobs=args.jobs)
> > + builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
> > + theme=args.theme, css=args.css, paper=args.paper)
> > +
> > +if __name__ == "__main__":
> > + main()
>
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-10 12:59 ` Mauro Carvalho Chehab
@ 2025-09-10 13:33 ` Mauro Carvalho Chehab
2025-09-12 10:15 ` Akira Yokosawa
2025-09-11 10:23 ` Jani Nikula
1 sibling, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-10 13:33 UTC (permalink / raw)
To: Jani Nikula
Cc: Jonathan Corbet, Linux Doc Mailing List, Björn Roy Baron,
Alex Gaynor, Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross,
linux-kernel, rust-for-linux
Em Wed, 10 Sep 2025 14:59:26 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:
> Em Wed, 10 Sep 2025 13:46:17 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
> > On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > > There are too much magic inside docs Makefile to properly run
> > > sphinx-build. Create an ancillary script that contains all
> > > kernel-related sphinx-build call logic currently at Makefile.
> > >
> > > Such script is designed to work both as an standalone command
> > > and as part of a Makefile. As such, it properly handles POSIX
> > > jobserver used by GNU make.
> > >
> > > On a side note, there was a line number increase due to the
> > > conversion:
> > >
> > > Documentation/Makefile | 131 +++----------
> > > tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++
> > > 2 files changed, 323 insertions(+), 101 deletions(-)
> > >
> > > This is because some things are more verbosed on Python and because
> > > it requires reading env vars from Makefile. Besides it, this script
> > > has some extra features that don't exist at the Makefile:
> > >
> > > - It can be called directly from command line;
> > > - It properly return PDF build errors.
> > >
> > > When running the script alone, it will only take handle sphinx-build
> > > targets. On other words, it won't runn make rustdoc after building
> > > htmlfiles, nor it will run the extra check scripts.
> >
> > I've always strongly believed we should aim to make it possible to build
> > the documentation by running sphinx-build directly on the
> > command-line. Not that it would be the common way to run it, but to not
> > accumulate things in the Makefile that need to happen before or
> > after. To promote handling the documentation build in Sphinx. To be able
> > to debug issues and try new Sphinx versions without all the hacks.
>
> That would be the better, but, unfortunately, this is not possible, for
> several reasons:
>
> 1. SPHINXDIRS. It needs a lot of magic to work, both before running
> sphinx-build and after (inside conf.py);
> 2. Several extensions require kernel-specific environment variables to
> work. Calling sphinx-build directly breaks them;
> 3. Sphinx itself doesn't build several targets alone. Instead, they create
> a Makefile, and an extra step is needed to finish the build. That's
> the case for pdf and texinfo, for instance;
> 4. Man pages generation. Sphinx support to generate it is very poor;
> 5. Rust integration adds more complexity to the table;
Heh, I ended forgetting what motivated me to do this work: the lack
of a native pdf builder (or an external one that actually works).
The current approach of using LaTeX for PDF is dirty:
- Sphinx can't produce a LaTeX file from the Kernel trees without
hundreds of warnings;
- latexmk hides some of them, but even it just one warning is reported,
the return status is not zero.
So, at the end of a PDF build via latex builder and the extra
makefile, there's no way to know if all PDF files were built or not,
except by having a somewhat complex logic that verifies all files
one by one.
We needed that to check if all PDF files were generated at the set of
test platforms we want docs build to work.
As the logic to check is complex, I would need to either add an
extra magic inside the already too complex Documentation/Makefile,
or to add one more hackish script.
Instead, I opted to reduce the number of scripts required during
PDF builds. So, this series:
- dropped the need of running jobserver;
- dropped the parallel jobs shell script;
- dropped the extra script to generate man pages;
- can later be integrated with sphinx-pre-install, dropping one
more script;
- didn't add an extra script to fix the return code for PDF;
- now summarizes what PDF files were actually generated and what
files weren't produced.
>
> I'm not seeing sphinx-build supporting the above needs anytime soon,
> and, even if we push our needs to Sphinx and it gets accepted there,
> we'll still need to wait for quite a while until LTS distros merge
> them.
>
> > This patch moves a bunch of that logic into a Python wrapper, and I feel
> > like it complicates matters. You can no longer rely on 'make V=1' to get
> > the build commands, for instance.
>
> Quite the opposite. if you try using "make V=1", it won't show the
> command line used to call sphinx-build anymore.
>
> This series restore it.
>
> See, if you build with this series with V=1, you will see exactly
> what commands are used on the build:
>
> $ make V=1 htmldocs
> ...
> python3 ./tools/docs/sphinx-build-wrapper htmldocs \
> --sphinxdirs="." --conf="conf.py" \
> --builddir="Documentation/output" \
> --theme= --css= --paper=
> python3 /new_devel/docs/sphinx_latest/bin/sphinx-build -j25 -b html -c /new_devel/docs/Documentation -d /new_devel/docs/Documentation/output/.doctrees -D kerneldoc_bin=scripts/kernel-doc.py -D version=6.17.0-rc1 -D release=6.17.0-rc1+ -D kerneldoc_srctree=. /new_devel/docs/Documentation /new_devel/docs/Documentation/output
> ...
>
>
>
> > Newer Sphinx versions have the -M option for "make mode". The Makefiles
> > produced by sphinx-quickstart only have one build target:
> >
> > # Catch-all target: route all unknown targets to Sphinx using the new
> > # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
>
> I didn't know about this, but from [1] it sounds it covers just two
> targets: "latexpdf" and "info".
>
> The most complex scenario is still not covered: SPHINXDIRS.
>
> [1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html
>
> > %: Makefile
> > @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
> >
> > That's all.
>
> Try doing such change on your makefile. it will break:
>
> - SPHINXDIRS;
> - V=1;
> - rustdoc
>
> and will still be dependent on variables that are passed via
> env from Kernel makefile. So, stil you can't run from command
> line. Also, if you call sphinx-build from command line:
>
> $ sphinx-build -j25 -b html Documentation Documentation/output
> ...
> File "<frozen os>", line 717, in __getitem__
> KeyError: 'srctree'
>
> It won't work, as several parameters that are required by conf.py and by
> Sphinx extensions would be missing (the most important one is srctree, but
> there are others in the line too).
>
> > The proposed wrapper duplicates loads of code that's supposed to be
> > handled by sphinx-build directly.
>
> Once we get the wrapper, we can work to simplify it, but still I
> can't see how to get rid of it.
>
> > Including the target/builder names.
>
> True, but this was a design decision taken lots of years ago: instead
> of:
> make html
>
> we're using:
>
> make htmldocs
>
> This series doesn't change that: either makefile or the script need
> to tho the namespace conversion.
>
> > Seems to me the goal should be to figure out *generic* wrappers for
> > handling parallelism, not Sphinx aware/specific.
> >
> >
> > BR,
> > Jani.
> >
> > >
> > > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > > ---
> > > Documentation/Makefile | 131 ++++----------
> > > tools/docs/sphinx-build-wrapper | 293 ++++++++++++++++++++++++++++++++
> > > 2 files changed, 323 insertions(+), 101 deletions(-)
> > > create mode 100755 tools/docs/sphinx-build-wrapper
> > >
> > > diff --git a/Documentation/Makefile b/Documentation/Makefile
> > > index deb2029228ed..4736f02b6c9e 100644
> > > --- a/Documentation/Makefile
> > > +++ b/Documentation/Makefile
> > > @@ -23,21 +23,22 @@ SPHINXOPTS =
> > > SPHINXDIRS = .
> > > DOCS_THEME =
> > > DOCS_CSS =
> > > -_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
> > > SPHINX_CONF = conf.py
> > > PAPER =
> > > BUILDDIR = $(obj)/output
> > > PDFLATEX = xelatex
> > > LATEXOPTS = -interaction=batchmode -no-shell-escape
> > >
> > > +PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
> > > +
> > > +# Wrapper for sphinx-build
> > > +
> > > +BUILD_WRAPPER = $(srctree)/tools/docs/sphinx-build-wrapper
> > > +
> > > # For denylisting "variable font" files
> > > # Can be overridden by setting as an env variable
> > > FONTS_CONF_DENY_VF ?= $(HOME)/deny-vf
> > >
> > > -ifeq ($(findstring 1, $(KBUILD_VERBOSE)),)
> > > -SPHINXOPTS += "-q"
> > > -endif
> > > -
> > > # User-friendly check for sphinx-build
> > > HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)
> > >
> > > @@ -51,63 +52,31 @@ ifeq ($(HAVE_SPHINX),0)
> > >
> > > else # HAVE_SPHINX
> > >
> > > -# User-friendly check for pdflatex and latexmk
> > > -HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
> > > -HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
> > > +# Common documentation targets
> > > +infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
> > > + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> > > + --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> > > + --builddir="$(BUILDDIR)" \
> > > + --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
> > >
> > > -ifeq ($(HAVE_LATEXMK),1)
> > > - PDFLATEX := latexmk -$(PDFLATEX)
> > > -endif #HAVE_LATEXMK
> > > -
> > > -# Internal variables.
> > > -PAPEROPT_a4 = -D latex_elements.papersize=a4paper
> > > -PAPEROPT_letter = -D latex_elements.papersize=letterpaper
> > > -ALLSPHINXOPTS = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
> > > -ALLSPHINXOPTS += $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
> > > -ifneq ($(wildcard $(srctree)/.config),)
> > > -ifeq ($(CONFIG_RUST),y)
> > > - # Let Sphinx know we will include rustdoc
> > > - ALLSPHINXOPTS += -t rustdoc
> > > -endif
> > > +# Special handling for pdfdocs
> > > +ifeq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0)
> > > +pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
> > > +else
> > > +pdfdocs:
> > > + $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> > > + @echo " SKIP Sphinx $@ target."
> > > endif
> > > -# the i18n builder cannot share the environment and doctrees with the others
> > > -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
> > > -
> > > -# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
> > > -loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
> > > -
> > > -# $2 sphinx builder e.g. "html"
> > > -# $3 name of the build subfolder / e.g. "userspace-api/media", used as:
> > > -# * dest folder relative to $(BUILDDIR) and
> > > -# * cache folder relative to $(BUILDDIR)/.doctrees
> > > -# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man
> > > -# $5 reST source folder relative to $(src),
> > > -# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentation/userspace-api/media
> > > -
> > > -PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
> > > -
> > > -quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
> > > - cmd_sphinx = \
> > > - PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
> > > - BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \
> > > - $(PYTHON3) $(srctree)/scripts/jobserver-exec \
> > > - $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
> > > - $(SPHINXBUILD) \
> > > - -b $2 \
> > > - -c $(abspath $(src)) \
> > > - -d $(abspath $(BUILDDIR)/.doctrees/$3) \
> > > - -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
> > > - $(ALLSPHINXOPTS) \
> > > - $(abspath $(src)/$5) \
> > > - $(abspath $(BUILDDIR)/$3/$4) && \
> > > - if [ "x$(DOCS_CSS)" != "x" ]; then \
> > > - cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(DOCS_CSS)) $(BUILDDIR)/$3/_static/; \
> > > - fi
> > >
> > > +# HTML main logic is identical to other targets. However, if rust is enabled,
> > > +# an extra step at the end is required to generate rustdoc.
> > > htmldocs:
> > > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
> > > -
> > > + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
> > > + --sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
> > > + --builddir="$(BUILDDIR)" \
> > > + --theme=$(DOCS_THEME) --css=$(DOCS_CSS) --paper=$(PAPER)
> > > # If Rust support is available and .config exists, add rustdoc generated contents.
> > > # If there are any, the errors from this make rustdoc will be displayed but
> > > # won't stop the execution of htmldocs
> > > @@ -118,49 +87,6 @@ ifeq ($(CONFIG_RUST),y)
> > > endif
> > > endif
> > >
> > > -texinfodocs:
> > > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
> > > -
> > > -# Note: the 'info' Make target is generated by sphinx itself when
> > > -# running the texinfodocs target define above.
> > > -infodocs: texinfodocs
> > > - $(MAKE) -C $(BUILDDIR)/texinfo info
> > > -
> > > -linkcheckdocs:
> > > - @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
> > > -
> > > -latexdocs:
> > > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
> > > -
> > > -ifeq ($(HAVE_PDFLATEX),0)
> > > -
> > > -pdfdocs:
> > > - $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
> > > - @echo " SKIP Sphinx $@ target."
> > > -
> > > -else # HAVE_PDFLATEX
> > > -
> > > -pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
> > > -pdfdocs: latexdocs
> > > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > - $(foreach var,$(SPHINXDIRS), \
> > > - $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
> > > - mkdir -p $(BUILDDIR)/$(var)/pdf; \
> > > - mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
> > > - )
> > > -
> > > -endif # HAVE_PDFLATEX
> > > -
> > > -epubdocs:
> > > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
> > > -
> > > -xmldocs:
> > > - @$(srctree)/tools/docs/sphinx-pre-install --version-check
> > > - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
> > > -
> > > endif # HAVE_SPHINX
> > >
> > > # The following targets are independent of HAVE_SPHINX, and the rules should
> > > @@ -172,6 +98,9 @@ refcheckdocs:
> > > cleandocs:
> > > $(Q)rm -rf $(BUILDDIR)
> > >
> > > +# Used only on help
> > > +_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
> > > +
> > > dochelp:
> > > @echo ' Linux kernel internal documentation in different formats from ReST:'
> > > @echo ' htmldocs - HTML'
> > > diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
> > > new file mode 100755
> > > index 000000000000..3256418d8dc5
> > > --- /dev/null
> > > +++ b/tools/docs/sphinx-build-wrapper
> > > @@ -0,0 +1,293 @@
> > > +#!/usr/bin/env python3
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +import argparse
> > > +import os
> > > +import shlex
> > > +import shutil
> > > +import subprocess
> > > +import sys
> > > +from lib.python_version import PythonVersion
> > > +
> > > +LIB_DIR = "../../scripts/lib"
> > > +SRC_DIR = os.path.dirname(os.path.realpath(__file__))
> > > +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
> > > +
> > > +from jobserver import JobserverExec
> > > +
> > > +MIN_PYTHON_VERSION = PythonVersion("3.7").version
> > > +PAPER = ["", "a4", "letter"]
> > > +TARGETS = {
> > > + "cleandocs": { "builder": "clean" },
> > > + "linkcheckdocs": { "builder": "linkcheck" },
> > > + "htmldocs": { "builder": "html" },
> > > + "epubdocs": { "builder": "epub", "out_dir": "epub" },
> > > + "texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" },
> > > + "infodocs": { "builder": "texinfo", "out_dir": "texinfo" },
> > > + "latexdocs": { "builder": "latex", "out_dir": "latex" },
> > > + "pdfdocs": { "builder": "latex", "out_dir": "latex" },
> > > + "xmldocs": { "builder": "xml", "out_dir": "xml" },
> > > +}
> > > +
> > > +class SphinxBuilder:
> > > + def is_rust_enabled(self):
> > > + config_path = os.path.join(self.srctree, ".config")
> > > + if os.path.isfile(config_path):
> > > + with open(config_path, "r", encoding="utf-8") as f:
> > > + return "CONFIG_RUST=y" in f.read()
> > > + return False
> > > +
> > > + def get_path(self, path, use_cwd=False, abs_path=False):
> > > + path = os.path.expanduser(path)
> > > + if not path.startswith("/"):
> > > + if use_cwd:
> > > + base = os.getcwd()
> > > + else:
> > > + base = self.srctree
> > > + path = os.path.join(base, path)
> > > + if abs_path:
> > > + return os.path.abspath(path)
> > > + return path
> > > +
> > > + def __init__(self, builddir, verbose=False, n_jobs=None):
> > > + self.verbose = None
> > > + self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
> > > + self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
> > > + self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
> > > + self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
> > > + if not verbose:
> > > + verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
> > > + if verbose is not None:
> > > + self.verbose = verbose
> > > + parser = argparse.ArgumentParser()
> > > + parser.add_argument('-j', '--jobs', type=int)
> > > + parser.add_argument('-q', '--quiet', type=int)
> > > + sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", ""))
> > > + sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
> > > + if sphinx_args.quiet is True:
> > > + self.verbose = False
> > > + if sphinx_args.jobs:
> > > + self.n_jobs = sphinx_args.jobs
> > > + self.n_jobs = n_jobs
> > > + self.srctree = os.environ.get("srctree")
> > > + if not self.srctree:
> > > + self.srctree = "."
> > > + os.environ["srctree"] = self.srctree
> > > + self.sphinxbuild = os.environ.get("SPHINXBUILD", "sphinx-build")
> > > + self.kerneldoc = self.get_path(os.environ.get("KERNELDOC",
> > > + "scripts/kernel-doc.py"))
> > > + self.builddir = self.get_path(builddir, use_cwd=True, abs_path=True)
> > > +
> > > + self.config_rust = self.is_rust_enabled()
> > > +
> > > + self.pdflatex_cmd = shutil.which(self.pdflatex)
> > > + self.latexmk_cmd = shutil.which("latexmk")
> > > +
> > > + self.env = os.environ.copy()
> > > +
> > > + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
> > > + with JobserverExec() as jobserver:
> > > + if jobserver.claim:
> > > + n_jobs = str(jobserver.claim)
> > > + else:
> > > + n_jobs = "auto" # Supported since Sphinx 1.7
> > > + cmd = []
> > > + cmd.append(sys.executable)
> > > + cmd.append(sphinx_build)
> > > + if self.n_jobs:
> > > + n_jobs = str(self.n_jobs)
> > > +
> > > + if n_jobs:
> > > + cmd += [f"-j{n_jobs}"]
> > > +
> > > + if not self.verbose:
> > > + cmd.append("-q")
> > > + cmd += self.sphinxopts
> > > + cmd += build_args
> > > + if self.verbose:
> > > + print(" ".join(cmd))
> > > + return subprocess.call(cmd, *args, **pwargs)
> > > +
> > > + def handle_html(self, css, output_dir):
> > > + if not css:
> > > + return
> > > + css = os.path.expanduser(css)
> > > + if not css.startswith("/"):
> > > + css = os.path.join(self.srctree, css)
> > > + static_dir = os.path.join(output_dir, "_static")
> > > + os.makedirs(static_dir, exist_ok=True)
> > > + try:
> > > + shutil.copy2(css, static_dir)
> > > + except (OSError, IOError) as e:
> > > + print(f"Warning: Failed to copy CSS: {e}", file=sys.stderr)
> > > +
> > > + def handle_pdf(self, output_dirs):
> > > + builds = {}
> > > + max_len = 0
> > > + for from_dir in output_dirs:
> > > + pdf_dir = os.path.join(from_dir, "../pdf")
> > > + os.makedirs(pdf_dir, exist_ok=True)
> > > + if self.latexmk_cmd:
> > > + latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
> > > + else:
> > > + latex_cmd = [self.pdflatex]
> > > + latex_cmd.extend(shlex.split(self.latexopts))
> > > + tex_suffix = ".tex"
> > > + has_tex = False
> > > + build_failed = False
> > > + with os.scandir(from_dir) as it:
> > > + for entry in it:
> > > + if not entry.name.endswith(tex_suffix):
> > > + continue
> > > + name = entry.name[:-len(tex_suffix)]
> > > + has_tex = True
> > > + try:
> > > + subprocess.run(latex_cmd + [entry.path],
> > > + cwd=from_dir, check=True)
> > > + except subprocess.CalledProcessError:
> > > + pass
> > > + pdf_name = name + ".pdf"
> > > + pdf_from = os.path.join(from_dir, pdf_name)
> > > + pdf_to = os.path.join(pdf_dir, pdf_name)
> > > + if os.path.exists(pdf_from):
> > > + os.rename(pdf_from, pdf_to)
> > > + builds[name] = os.path.relpath(pdf_to, self.builddir)
> > > + else:
> > > + builds[name] = "FAILED"
> > > + build_failed = True
> > > + name = entry.name.removesuffix(".tex")
> > > + max_len = max(max_len, len(name))
> > > +
> > > + if not has_tex:
> > > + name = os.path.basename(from_dir)
> > > + max_len = max(max_len, len(name))
> > > + builds[name] = "FAILED (no .tex)"
> > > + build_failed = True
> > > + msg = "Summary"
> > > + msg += "\n" + "=" * len(msg)
> > > + print()
> > > + print(msg)
> > > + for pdf_name, pdf_file in builds.items():
> > > + print(f"{pdf_name:<{max_len}}: {pdf_file}")
> > > + print()
> > > + if build_failed:
> > > + sys.exit("PDF build failed: not all PDF files were created.")
> > > + else:
> > > + print("All PDF files were built.")
> > > +
> > > + def handle_info(self, output_dirs):
> > > + for output_dir in output_dirs:
> > > + try:
> > > + subprocess.run(["make", "info"], cwd=output_dir, check=True)
> > > + except subprocess.CalledProcessError as e:
> > > + sys.exit(f"Error generating info docs: {e}")
> > > +
> > > + def cleandocs(self, builder):
> > > + shutil.rmtree(self.builddir, ignore_errors=True)
> > > +
> > > + def build(self, target, sphinxdirs=None, conf="conf.py",
> > > + theme=None, css=None, paper=None):
> > > + builder = TARGETS[target]["builder"]
> > > + out_dir = TARGETS[target].get("out_dir", "")
> > > + if target == "cleandocs":
> > > + self.cleandocs(builder)
> > > + return
> > > + if theme:
> > > + os.environ["DOCS_THEME"] = theme
> > > + sphinxbuild = shutil.which(self.sphinxbuild, path=self.env["PATH"])
> > > + if not sphinxbuild:
> > > + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n")
> > > + if builder == "latex":
> > > + if not self.pdflatex_cmd and not self.latexmk_cmd:
> > > + sys.exit("Error: pdflatex or latexmk required for PDF generation")
> > > + docs_dir = os.path.abspath(os.path.join(self.srctree, "Documentation"))
> > > + kerneldoc = self.kerneldoc
> > > + if kerneldoc.startswith(self.srctree):
> > > + kerneldoc = os.path.relpath(kerneldoc, self.srctree)
> > > + args = [ "-b", builder, "-c", docs_dir ]
> > > + if builder == "latex":
> > > + if not paper:
> > > + paper = PAPER[1]
> > > + args.extend(["-D", f"latex_elements.papersize={paper}paper"])
> > > + if self.config_rust:
> > > + args.extend(["-t", "rustdoc"])
> > > + if conf:
> > > + self.env["SPHINX_CONF"] = self.get_path(conf, abs_path=True)
> > > + if not sphinxdirs:
> > > + sphinxdirs = os.environ.get("SPHINXDIRS", ".")
> > > + sphinxdirs_list = []
> > > + for sphinxdir in sphinxdirs:
> > > + if isinstance(sphinxdir, list):
> > > + sphinxdirs_list += sphinxdir
> > > + else:
> > > + for name in sphinxdir.split(" "):
> > > + sphinxdirs_list.append(name)
> > > + output_dirs = []
> > > + for sphinxdir in sphinxdirs_list:
> > > + src_dir = os.path.join(docs_dir, sphinxdir)
> > > + doctree_dir = os.path.join(self.builddir, ".doctrees")
> > > + output_dir = os.path.join(self.builddir, sphinxdir, out_dir)
> > > + src_dir = os.path.normpath(src_dir)
> > > + doctree_dir = os.path.normpath(doctree_dir)
> > > + output_dir = os.path.normpath(output_dir)
> > > + os.makedirs(doctree_dir, exist_ok=True)
> > > + os.makedirs(output_dir, exist_ok=True)
> > > + output_dirs.append(output_dir)
> > > + build_args = args + [
> > > + "-d", doctree_dir,
> > > + "-D", f"kerneldoc_bin={kerneldoc}",
> > > + "-D", f"version={self.kernelversion}",
> > > + "-D", f"release={self.kernelrelease}",
> > > + "-D", f"kerneldoc_srctree={self.srctree}",
> > > + src_dir,
> > > + output_dir,
> > > + ]
> > > + try:
> > > + self.run_sphinx(sphinxbuild, build_args, env=self.env)
> > > + except (OSError, ValueError, subprocess.SubprocessError) as e:
> > > + sys.exit(f"Build failed: {repr(e)}")
> > > + if target in ["htmldocs", "epubdocs"]:
> > > + self.handle_html(css, output_dir)
> > > + if target == "pdfdocs":
> > > + self.handle_pdf(output_dirs)
> > > + elif target == "infodocs":
> > > + self.handle_info(output_dirs)
> > > +
> > > +def jobs_type(value):
> > > + if value is None:
> > > + return None
> > > + if value.lower() == 'auto':
> > > + return value.lower()
> > > + try:
> > > + if int(value) >= 1:
> > > + return value
> > > + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}")
> > > + except ValueError:
> > > + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}")
> > > +
> > > +def main():
> > > + parser = argparse.ArgumentParser(description="Kernel documentation builder")
> > > + parser.add_argument("target", choices=list(TARGETS.keys()),
> > > + help="Documentation target to build")
> > > + parser.add_argument("--sphinxdirs", nargs="+",
> > > + help="Specific directories to build")
> > > + parser.add_argument("--conf", default="conf.py",
> > > + help="Sphinx configuration file")
> > > + parser.add_argument("--builddir", default="output",
> > > + help="Sphinx configuration file")
> > > + parser.add_argument("--theme", help="Sphinx theme to use")
> > > + parser.add_argument("--css", help="Custom CSS file for HTML/EPUB")
> > > + parser.add_argument("--paper", choices=PAPER, default=PAPER[0],
> > > + help="Paper size for LaTeX/PDF output")
> > > + parser.add_argument("-v", "--verbose", action='store_true',
> > > + help="place build in verbose mode")
> > > + parser.add_argument('-j', '--jobs', type=jobs_type,
> > > + help="Sets number of jobs to use with sphinx-build")
> > > + args = parser.parse_args()
> > > + PythonVersion.check_python(MIN_PYTHON_VERSION)
> > > + builder = SphinxBuilder(builddir=args.builddir,
> > > + verbose=args.verbose, n_jobs=args.jobs)
> > > + builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
> > > + theme=args.theme, css=args.css, paper=args.paper)
> > > +
> > > +if __name__ == "__main__":
> > > + main()
> >
>
>
>
> Thanks,
> Mauro
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-10 13:33 ` Mauro Carvalho Chehab
@ 2025-09-12 10:15 ` Akira Yokosawa
2025-09-12 11:04 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Akira Yokosawa @ 2025-09-12 10:15 UTC (permalink / raw)
To: mchehab+huawei
Cc: corbet, jani.nikula, linux-doc, linux-kernel, tmgross,
Akira Yokosawa
[-CC: rust people and list]
OK, Looks like I have to bite.
On Wed, 10 Sep 2025 15:33:34 +0200, Mauro Carvalho Chehab wrote:
[...]
> The current approach of using LaTeX for PDF is dirty:
>
> - Sphinx can't produce a LaTeX file from the Kernel trees without
> hundreds of warnings;
> - latexmk hides some of them, but even it just one warning is reported,
> the return status is not zero.
This sounds interesting to me.
As far I remember, I have never seen such strange results of latexmk
under build envs where all the necessary packages are properly installed.
I think what you are trying here is to paper over whatever bug in latexmk/
xelatex by always ignoring their exit status. Am I guessing right?
If that is the case, I'd rather report such an issue at upstream
lagtexmk/xelatex.
Can you please provide a reproducer of such an issue?
Or you saw something while you were tackling issues you claimed in the
cover letter [1] of "Fix PDF doc builds on major distros" series?
[1]: https://lore.kernel.org/cover.1755763127.git.mchehab+huawei@kernel.org/
Thanks,
Akira
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 10:15 ` Akira Yokosawa
@ 2025-09-12 11:04 ` Mauro Carvalho Chehab
2025-09-12 14:03 ` Akira Yokosawa
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 11:04 UTC (permalink / raw)
To: Akira Yokosawa; +Cc: corbet, jani.nikula, linux-doc, linux-kernel, tmgross
Em Fri, 12 Sep 2025 19:15:44 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:
> [-CC: rust people and list]
>
> OK, Looks like I have to bite.
>
> On Wed, 10 Sep 2025 15:33:34 +0200, Mauro Carvalho Chehab wrote:
> [...]
>
> > The current approach of using LaTeX for PDF is dirty:
> >
> > - Sphinx can't produce a LaTeX file from the Kernel trees without
> > hundreds of warnings;
> > - latexmk hides some of them, but even it just one warning is reported,
> > the return status is not zero.
>
> This sounds interesting to me.
> As far I remember, I have never seen such strange results of latexmk
> under build envs where all the necessary packages are properly installed.
I saw it here on multiple distros including Fedora (which is the one
I use on my desktop). Media jenkins CI running on Debian always suffered
from such issues, up to the point I started ignoring pdf build results.
> I think what you are trying here is to paper over whatever bug in latexmk/
> xelatex by always ignoring their exit status. Am I guessing right?
>
> If that is the case, I'd rather report such an issue at upstream
> lagtexmk/xelatex.
I'm not entirely sure if this is a bug or a feature. Last time I got
one such error and did a web search, I saw similar reports explaining
the error code as if it is an expected behavior.
This is a known bug, and the current building system has a poor man
workaround for it already:
pdfdocs: latexdocs
@$(srctree)/scripts/sphinx-pre-install --version-check
$(foreach var,$(SPHINXDIRS), \
$(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
mkdir -p $(BUILDDIR)/$(var)/pdf; \
mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
)
see the "||" at make pipeline. The final "|| exit" makes the build
ignore pdf build errors. It is there since the beginning:
cb43fb5775df ("docs: remove DocBook from the building system")
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 11:04 ` Mauro Carvalho Chehab
@ 2025-09-12 14:03 ` Akira Yokosawa
2025-09-12 14:50 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Akira Yokosawa @ 2025-09-12 14:03 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: corbet, jani.nikula, linux-doc, linux-kernel, tmgross
On Fri, 12 Sep 2025 13:04:20 +0200, Mauro Carvalho Chehab wrote:
> Em Fri, 12 Sep 2025 19:15:44 +0900
> Akira Yokosawa <akiyks@gmail.com> escreveu:
>
>> [-CC: rust people and list]
>>
>> OK, Looks like I have to bite.
>>
>> On Wed, 10 Sep 2025 15:33:34 +0200, Mauro Carvalho Chehab wrote:
>> [...]
>>
>>> The current approach of using LaTeX for PDF is dirty:
>>>
>>> - Sphinx can't produce a LaTeX file from the Kernel trees without
>>> hundreds of warnings;
>>> - latexmk hides some of them, but even it just one warning is reported,
>>> the return status is not zero.
>>
>> This sounds interesting to me.
>> As far I remember, I have never seen such strange results of latexmk
>> under build envs where all the necessary packages are properly installed.
>
> I saw it here on multiple distros including Fedora (which is the one
> I use on my desktop). Media jenkins CI running on Debian always suffered
> from such issues, up to the point I started ignoring pdf build results.
>
So please provide exact steps for me to see such errors.
I don't have any issues after strictly following the suggestions from
sphinx-pre-install under Fedora.
I even invoked [...]/output/latex/Makefile manually after running
"make latexdocs" by:
- cd [...]/output/latex/
- make PDFLATEX="latexmk -xelatex" LATEXOPTS="-interaction=batchmode -no-shell-escape" -j6 -O all
, and all the PDFs were built without any issues.
Quite puzzling ...
Or does your Fedora have some Noto CJK variable fonts installed?
Hmm ...
Thanks,
Akira
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 14:03 ` Akira Yokosawa
@ 2025-09-12 14:50 ` Mauro Carvalho Chehab
2025-09-15 8:27 ` Akira Yokosawa
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 14:50 UTC (permalink / raw)
To: Akira Yokosawa
Cc: Mauro Carvalho Chehab, corbet, jani.nikula, linux-doc,
linux-kernel, tmgross
On Fri, Sep 12, 2025 at 11:03:43PM +0900, Akira Yokosawa wrote:
> On Fri, 12 Sep 2025 13:04:20 +0200, Mauro Carvalho Chehab wrote:
> > Em Fri, 12 Sep 2025 19:15:44 +0900
> > Akira Yokosawa <akiyks@gmail.com> escreveu:
> >
> >> [-CC: rust people and list]
> >>
> >> OK, Looks like I have to bite.
> >>
> >> On Wed, 10 Sep 2025 15:33:34 +0200, Mauro Carvalho Chehab wrote:
> >> [...]
> >>
> >>> The current approach of using LaTeX for PDF is dirty:
> >>>
> >>> - Sphinx can't produce a LaTeX file from the Kernel trees without
> >>> hundreds of warnings;
> >>> - latexmk hides some of them, but even it just one warning is reported,
> >>> the return status is not zero.
> >>
> >> This sounds interesting to me.
> >> As far I remember, I have never seen such strange results of latexmk
> >> under build envs where all the necessary packages are properly installed.
> >
> > I saw it here on multiple distros including Fedora (which is the one
> > I use on my desktop). Media jenkins CI running on Debian always suffered
> > from such issues, up to the point I started ignoring pdf build results.
> >
>
> So please provide exact steps for me to see such errors.
Sorry, but I don't have enough time to try reproducing it again
(plus, I'm ran out of disk space on my /var partition forcing me to
reclaim the space used by my test containers).
>
> I don't have any issues after strictly following the suggestions from
> sphinx-pre-install under Fedora.
>
> I even invoked [...]/output/latex/Makefile manually after running
> "make latexdocs" by:
>
> - cd [...]/output/latex/
> - make PDFLATEX="latexmk -xelatex" LATEXOPTS="-interaction=batchmode -no-shell-escape" -j6 -O all
>
> , and all the PDFs were built without any issues.
>
> Quite puzzling ...
>
> Or does your Fedora have some Noto CJK variable fonts installed?
On my main desktop, yes, that's the case: it currently has some
Noto CJK fonts installed. I wasn't aware about such issues
with Fedora until today, when I noticed your check script.
On my test containers, all docs were built even on Fedora.
Yet, the issue that forced us to add "|| exit" hack to ignore PDF
build errors is not new. As I pointed, at least since 2017 we
have a hack due to that our Makefile, but I guess we had an older
hack as well.
I dunno the exact conditions, but depending on latex version, distro,
if the computer is in bad mood, if it is rainning - or whavever other
random condition - even when all PDF docs are built, make pdf inside
output/latex may return non-zero, for warnings. Maybe it could be
also related of using latexmk or calling xelatex directly.
If I recall corretly, we added latexmk to fix some of such build
issues.
-
In any case, this changeset fix it on several ways:
- A failure while building one pdf doesn't prevent building other
files. With make, it may stop before building them all (if we drop
the "|| exit");
- It prints a summary reporting what PDF files were actually built,
so it is easy for the developer to know what broke (and it is
also easily parsed by a CI);
- The return code is zero only if all docs were built.
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 14:50 ` Mauro Carvalho Chehab
@ 2025-09-15 8:27 ` Akira Yokosawa
2025-09-15 10:58 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Akira Yokosawa @ 2025-09-15 8:27 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: corbet, jani.nikula, linux-doc, linux-kernel, tmgross,
Akira Yokosawa
On Fri, 12 Sep 2025 16:50:36 +0200, Mauro Carvalho Chehab wrote:
> On Fri, Sep 12, 2025 at 11:03:43PM +0900, Akira Yokosawa wrote:
>> On Fri, 12 Sep 2025 13:04:20 +0200, Mauro Carvalho Chehab wrote:
[...]
>>> I saw it here on multiple distros including Fedora (which is the one
>>> I use on my desktop). Media jenkins CI running on Debian always suffered
>>> from such issues, up to the point I started ignoring pdf build results.
>>>
>>
>> So please provide exact steps for me to see such errors.
>
> Sorry, but I don't have enough time to try reproducing it again
> (plus, I'm ran out of disk space on my /var partition forcing me to
> reclaim the space used by my test containers).
There is no urgency on my side. Please take your time.
>
>>
>> I don't have any issues after strictly following the suggestions from
>> sphinx-pre-install under Fedora.
>>
>> I even invoked [...]/output/latex/Makefile manually after running
>> "make latexdocs" by:
>>
>> - cd [...]/output/latex/
>> - make PDFLATEX="latexmk -xelatex" LATEXOPTS="-interaction=batchmode -no-shell-escape" -j6 -O all
>>
>> , and all the PDFs were built without any issues.
>>
>> Quite puzzling ...
>>
>> Or does your Fedora have some Noto CJK variable fonts installed?
>
> On my main desktop, yes, that's the case: it currently has some
> Noto CJK fonts installed. I wasn't aware about such issues
> with Fedora until today, when I noticed your check script.
>
Good. That should be a step forward, I guess.
I know you have repeatedly said it is not the purpose of this series
to fix those issues in your images, but I have an impression that it is
closely related to testing sphinx-pre-install, and the objective of this
series is to make the testing/debugging of sphinx-pre-install easier for
you.
But, at least for me, the behavior you want for "pdfdocs" is not
ideal for regular testing of .rst and kernel-doc changes in kernel
source side. I want "make pdfdocs" to give up earlier rather than later.
It should leave relevant info near the bottom of terminal log.
Now, here are my responses to your arguments:
> In any case, this changeset fix it on several ways:
>
> - A failure while building one pdf doesn't prevent building other
> files. With make, it may stop before building them all (if we drop
> the "|| exit");
Didn't you mean "(if we keep the "|| exit"); ???
If you drop the "|| exit", which will cause false-negatives.
And you are going to test every PDFs at the final stage of pdfdocs
to catch such false-positives.
Sounds like a plan.
> - It prints a summary reporting what PDF files were actually built,
> so it is easy for the developer to know what broke (and it is
> also easily parsed by a CI);
> - The return code is zero only if all docs were built.
I agree this is an improvement, but if we get rid of the loop construct
in the Makefile, we can forget about said false-negatives, can't we?
How about something like the following approach?
Let's think of SPHINXDIRS="admin-guide core-api driver-api userspace-api"
In this case "make latexdocs" will generate
output/admin-guide/latex/
output/core-api/latex/
output/driver-api/latex/
output/userspace-api/latex/
They will have identical boiler-plate files latex builder would emit,
and subdir-specific files such as:
output/admin-guide/latex/admin-guide.tex
c3-isp.dot
c3-isp.pdf
c3-isp.svg
conn-states-8.dot
conn-states-8.pdf
conn-states-8.svg
disk-states-8.dot
disk-states-8.pdf
disk-states-8.svg
...
output/core-api/latex/core-api.tex
output/driver-api/latex/driver-api.tex
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.dot
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.svg
DOT-289c17ebc0291f90ccaf431961707504464a78d4.dot
DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf
DOT-289c17ebc0291f90ccaf431961707504464a78d4.svg
...
output/userspace-api/latex/userspace-api.tex
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.dot
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.svg
DOT-289c17ebc0291f90ccaf431961707504464a78d4.dot
DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf
DOT-289c17ebc0291f90ccaf431961707504464a78d4.svg
...
At a pre-processing stage of pdfdocs, you create output/latex/ and
symlink subdir-specific files needed for latexmk/xelatex into there.
(Copying them should work too.)
output/latex/admin-guide.tex --> ../admin-guide/latex/
c3-isp.pdf --> ../admin-guide/latex/
conn-states-8.pdf --> ../admin-guide/latex/
disk-states-8.pdf --> ../admin-guide/latex/
...
core-api.tex --> ../core-api/latex/
driver-api.tex --> ../driver-api/latex/
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf --> ../driver-api/latex/
DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf --> ../driver-api/latex/
...
userspace-api.tex --> ../userspace-api/
DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf --> ../userspace-api/latex/
DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf --> ../userspace-api/latex/
...
The latexmk stage would now be identical to the SPHINXDIRS="." case,
meaning it won't need the loop in the recipe.
Furthermore, post-processing would be almost the same as the default
case.
As a bonus, "make -j -O" will work as the same as full pdfdocs build.
If you are interested, I think I can prepare a PoC patch, probably
after v6.18-rc1.
If your sphinx-build-wrapper's latexdocs stage can be adjusted so that
said pre-processing of pdfdocs can be made unnecessary, that would be
even better.
Regards,
Akira
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-15 8:27 ` Akira Yokosawa
@ 2025-09-15 10:58 ` Mauro Carvalho Chehab
2025-09-15 12:54 ` Jani Nikula
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-15 10:58 UTC (permalink / raw)
To: Akira Yokosawa; +Cc: corbet, jani.nikula, linux-doc, linux-kernel, tmgross
Em Mon, 15 Sep 2025 17:27:17 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:
> On Fri, 12 Sep 2025 16:50:36 +0200, Mauro Carvalho Chehab wrote:
> > On Fri, Sep 12, 2025 at 11:03:43PM +0900, Akira Yokosawa wrote:
> >> On Fri, 12 Sep 2025 13:04:20 +0200, Mauro Carvalho Chehab wrote:
> [...]
>
> >>> I saw it here on multiple distros including Fedora (which is the one
> >>> I use on my desktop). Media jenkins CI running on Debian always suffered
> >>> from such issues, up to the point I started ignoring pdf build results.
> >>>
> >>
> >> So please provide exact steps for me to see such errors.
> >
> > Sorry, but I don't have enough time to try reproducing it again
> > (plus, I'm ran out of disk space on my /var partition forcing me to
> > reclaim the space used by my test containers).
>
> There is no urgency on my side. Please take your time.
>
> >
> >>
> >> I don't have any issues after strictly following the suggestions from
> >> sphinx-pre-install under Fedora.
> >>
> >> I even invoked [...]/output/latex/Makefile manually after running
> >> "make latexdocs" by:
> >>
> >> - cd [...]/output/latex/
> >> - make PDFLATEX="latexmk -xelatex" LATEXOPTS="-interaction=batchmode -no-shell-escape" -j6 -O all
> >>
> >> , and all the PDFs were built without any issues.
> >>
> >> Quite puzzling ...
> >>
> >> Or does your Fedora have some Noto CJK variable fonts installed?
> >
> > On my main desktop, yes, that's the case: it currently has some
> > Noto CJK fonts installed. I wasn't aware about such issues
> > with Fedora until today, when I noticed your check script.
> >
>
> Good. That should be a step forward, I guess.
>
> I know you have repeatedly said it is not the purpose of this series
> to fix those issues in your images, but I have an impression that it is
> closely related to testing sphinx-pre-install, and the objective of this
> series is to make the testing/debugging of sphinx-pre-install easier for
> you.
No, that's not the case. Yes, it helped me to test the script, but the
goal here is completely unrelated, and it actually solves a problem
I suffered a lot with CI jobs: I do want to know as soon as possible
when a patch to media breaks uAPI and/or kAPI books. I had a CI job
running to test such cases here:
https://builder.linuxtv.org/view/Kernel/job/media.git_drivers_build/
and on other pipelines, but I had to disable and/or ignore PDF builds,
with ended culminating with that "|| exit" line on docs makefile, as
this was causing CI to always report errors, even when PDF was successfully
built.
For such purpose, having a reliable SUCCESS return code when the docs
are built is a must.
> But, at least for me, the behavior you want for "pdfdocs" is not
> ideal for regular testing of .rst and kernel-doc changes in kernel
> source side. I want "make pdfdocs" to give up earlier rather than later.
> It should leave relevant info near the bottom of terminal log.
For me, the most important data is what documents broke between
two builds. That's why at the end I want to have the failed docs.
Yet, I guess we can have a solution that may be satisfying what
you want and what I want: only print success when V=1. When V=1,
it can just print the failed ones at the end.
> Now, here are my responses to your arguments:
>
> > In any case, this changeset fix it on several ways:
> >
> > - A failure while building one pdf doesn't prevent building other
> > files. With make, it may stop before building them all (if we drop
> > the "|| exit");
>
> Didn't you mean "(if we keep the "|| exit"); ???
>
> If you drop the "|| exit", which will cause false-negatives.
> And you are going to test every PDFs at the final stage of pdfdocs
> to catch such false-positives.
>
> Sounds like a plan.
No, it doesn't.
Dropping it causes false-positives. That's the root cause why we
had to add it in the first place. Try looking at latexmk and xelatex
man pages: there are no descriptions about what they're supposed
to return, and, on practical experiments, return code == 0 doesn't mean
that all PDF builds worked. It means something else.
Heh, looking at latexmk, besides the "explicit" codes it has, it seems
it just returns whatever xelatex returns.
Also, if you see, for instance:
https://math.nist.gov/~BMiller/LaTeXML/manual/errorcodes/
Latex has 3 types of errors:
Warnings
Errors
Fatals
Only "Fatals" are certain to cause PDF build failures. "Errors"
can be a lot of things and they usually are OK during Sphinx build,
as they're usually minor visual glitches.
I'm yet to find a document clearly describing what it returns
for each case, but I'm pretty sure trusting on xelatex return
code is the wrong thing to do: this never worked.
> > - It prints a summary reporting what PDF files were actually built,
> > so it is easy for the developer to know what broke (and it is
> > also easily parsed by a CI);
> > - The return code is zero only if all docs were built.
>
> I agree this is an improvement, but if we get rid of the loop construct
> in the Makefile, we can forget about said false-negatives, can't we?
No. See above.
> How about something like the following approach?
>
> Let's think of SPHINXDIRS="admin-guide core-api driver-api userspace-api"
>
> In this case "make latexdocs" will generate
>
> output/admin-guide/latex/
> output/core-api/latex/
> output/driver-api/latex/
> output/userspace-api/latex/
>
> They will have identical boiler-plate files latex builder would emit,
> and subdir-specific files such as:
>
> output/admin-guide/latex/admin-guide.tex
> c3-isp.dot
> c3-isp.pdf
> c3-isp.svg
> conn-states-8.dot
> conn-states-8.pdf
> conn-states-8.svg
> disk-states-8.dot
> disk-states-8.pdf
> disk-states-8.svg
> ...
> output/core-api/latex/core-api.tex
> output/driver-api/latex/driver-api.tex
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.dot
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.svg
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.dot
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.svg
> ...
> output/userspace-api/latex/userspace-api.tex
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.dot
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.svg
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.dot
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.svg
> ...
>
>
> At a pre-processing stage of pdfdocs, you create output/latex/ and
> symlink subdir-specific files needed for latexmk/xelatex into there.
> (Copying them should work too.)
>
> output/latex/admin-guide.tex --> ../admin-guide/latex/
> c3-isp.pdf --> ../admin-guide/latex/
> conn-states-8.pdf --> ../admin-guide/latex/
> disk-states-8.pdf --> ../admin-guide/latex/
> ...
> core-api.tex --> ../core-api/latex/
> driver-api.tex --> ../driver-api/latex/
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf --> ../driver-api/latex/
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf --> ../driver-api/latex/
> ...
> userspace-api.tex --> ../userspace-api/
> DOT-1e98886fceca6e25a115532f5efebb44c09dc98b.pdf --> ../userspace-api/latex/
> DOT-289c17ebc0291f90ccaf431961707504464a78d4.pdf --> ../userspace-api/latex/
> ...
>
> The latexmk stage would now be identical to the SPHINXDIRS="." case,
> meaning it won't need the loop in the recipe.
> Furthermore, post-processing would be almost the same as the default
> case.
This won't work, as we may have "duplicated" non-identical names, like:
SPHIHXDIRS="userspace-api/media Documentation/driver-api/media/"
both will produce media.tex files, with completely different contents.
> As a bonus, "make -j -O" will work as the same as full pdfdocs build.
This series doesn't break "make O=output_dir -j".
> If you are interested, I think I can prepare a PoC patch, probably
> after v6.18-rc1.
>
> If your sphinx-build-wrapper's latexdocs stage can be adjusted so that
> said pre-processing of pdfdocs can be made unnecessary, that would be
> even better.
It can be adjusted to whatever we want, provided that we find a
solution that works fine. It is a lot easier to do such kind
of changes in Python than in Makefile. At the plus side, adding
documentation to each step of the build process is easier.
IMHO, long term solution is to change SPHINXDIRS into something
like:
make O=doc_build SPHINXTITLE="Media docs" SPHINXDIRS="admin-guide/media userspace-api/media driver-api/media/"
would create something similar to this(*):
doc_build/sphindirs/
|
+--> index.rst
+--> admin-guide -> {srcdir}/Documentation/admin-guide/media/
+--> usespace-api -> {srcdir}/Documentation/admin-guide/media/
\--> driver-api -> {srcdir}/Documentation/admin-guide/media/
And then build, without any loops, using doc_build/sphindirs/ as
the sphinx-build sourcedir.
The problem of such approach is to avoid cross-references using
:doc:.
(*) this is oversimplified: it probably needs to copy
files, as Sphinx blocks symlinks like that. Also,
the actual linkname may be different - all we need
there is an unique name.
In any case, before we start looking into ways to avoid
SPHINXDIRS loop, IMO we'll be best served once we merge this
series and have the entire hacks on a single file without
depending on 4 independent scripts and relying on "|| exit"
hacks.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-15 10:58 ` Mauro Carvalho Chehab
@ 2025-09-15 12:54 ` Jani Nikula
2025-09-15 13:50 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-15 12:54 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Akira Yokosawa
Cc: corbet, linux-doc, linux-kernel, tmgross
On Mon, 15 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> IMHO, long term solution is to change SPHINXDIRS into something
> like:
>
> make O=doc_build SPHINXTITLE="Media docs" SPHINXDIRS="admin-guide/media userspace-api/media driver-api/media/"
>
> would create something similar to this(*):
>
> doc_build/sphindirs/
> |
> +--> index.rst
> +--> admin-guide -> {srcdir}/Documentation/admin-guide/media/
> +--> usespace-api -> {srcdir}/Documentation/admin-guide/media/
> \--> driver-api -> {srcdir}/Documentation/admin-guide/media/
So you're basically suggesting the documentation build should support
cherry-picking parts of the documentation with categories different from
what the upstream documentation has? I.e. even if we figured out how to
do intersphinx books, you'd want to grab parts from them and turn them
into something else?
Ugh.
BR,
Jani.
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-15 12:54 ` Jani Nikula
@ 2025-09-15 13:50 ` Mauro Carvalho Chehab
2025-09-15 14:33 ` Jani Nikula
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-15 13:50 UTC (permalink / raw)
To: Jani Nikula
Cc: Mauro Carvalho Chehab, Akira Yokosawa, corbet, linux-doc,
linux-kernel, tmgross
On Mon, Sep 15, 2025 at 03:54:26PM +0300, Jani Nikula wrote:
> On Mon, 15 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > IMHO, long term solution is to change SPHINXDIRS into something
> > like:
> >
> > make O=doc_build SPHINXTITLE="Media docs" SPHINXDIRS="admin-guide/media userspace-api/media driver-api/media/"
> >
> > would create something similar to this(*):
> >
> > doc_build/sphindirs/
> > |
> > +--> index.rst
> > +--> admin-guide -> {srcdir}/Documentation/admin-guide/media/
> > +--> usespace-api -> {srcdir}/Documentation/admin-guide/media/
> > \--> driver-api -> {srcdir}/Documentation/admin-guide/media/
>
> So you're basically suggesting the documentation build should support
> cherry-picking parts of the documentation with categories different from
> what the upstream documentation has?
No. I'm saying that, if we want to have a single build process
for multiple sphinxdirs, that sounds to be the better way to do it
to override sphinx-build limitation of having single source directory.
The advantages is that:
- brings more performance, as a single build would be enough;
- cross-references between them will be properly solved.
The disadvantages are:
- it would very likely need to create copies (or hard symlinks)
at the build dir, which may reduce performance;
- yet-another-hack;
- increased build complexity.
I'm not convinced myself about doing it or not. I didn't like when
I had to do that after the media book was split on 3 books. If one thinks
that having for loops to build targets is a problem, we need a separate
discussion about how to avoid it. Also, this is outside of the scope of
this series.
-
Another alternative to achieve such goal of not needing a loop at Sphinx
to handle multiple books in parallel would be to submit a patch for
Sphinx to get rid of the current limitation of having a single book
with everything on a single directory. Sphinx has already hacks for it
with "latex_documents", "man_pages", "texinfo_documents" conf.py variables
that are specific for non-html builders.
Still, when such variables are used, a post-sphinx-build logic with a
per-output-file loop is needed.
> I.e. even if we figured out how to
> do intersphinx books, you'd want to grab parts from them and turn them
> into something else?
Either doing it or not, intersphinx is intestesting.
> Ugh.
>
>
> BR,
> Jani.
>
>
> --
> Jani Nikula, Intel
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-15 13:50 ` Mauro Carvalho Chehab
@ 2025-09-15 14:33 ` Jani Nikula
2025-09-15 15:05 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-15 14:33 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Mauro Carvalho Chehab, Akira Yokosawa, corbet, linux-doc,
linux-kernel, tmgross
On Mon, 15 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> On Mon, Sep 15, 2025 at 03:54:26PM +0300, Jani Nikula wrote:
>> On Mon, 15 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> > IMHO, long term solution is to change SPHINXDIRS into something
>> > like:
>> >
>> > make O=doc_build SPHINXTITLE="Media docs" SPHINXDIRS="admin-guide/media userspace-api/media driver-api/media/"
>> >
>> > would create something similar to this(*):
>> >
>> > doc_build/sphindirs/
>> > |
>> > +--> index.rst
>> > +--> admin-guide -> {srcdir}/Documentation/admin-guide/media/
>> > +--> usespace-api -> {srcdir}/Documentation/admin-guide/media/
>> > \--> driver-api -> {srcdir}/Documentation/admin-guide/media/
>>
>> So you're basically suggesting the documentation build should support
>> cherry-picking parts of the documentation with categories different from
>> what the upstream documentation has?
>
> No. I'm saying that, if we want to have a single build process
> for multiple sphinxdirs, that sounds to be the better way to do it
> to override sphinx-build limitation of having single source directory.
>
> The advantages is that:
> - brings more performance, as a single build would be enough;
> - cross-references between them will be properly solved.
>
> The disadvantages are:
> - it would very likely need to create copies (or hard symlinks)
> at the build dir, which may reduce performance;
> - yet-another-hack;
> - increased build complexity.
>
> I'm not convinced myself about doing it or not. I didn't like when
> I had to do that after the media book was split on 3 books. If one thinks
> that having for loops to build targets is a problem, we need a separate
> discussion about how to avoid it. Also, this is outside of the scope of
> this series.
I honestly don't even understand what you're saying above, and how it
contradicts with what I said about cherry-picking the documentation to
build.
>
> -
>
> Another alternative to achieve such goal of not needing a loop at Sphinx
> to handle multiple books in parallel would be to submit a patch for
> Sphinx to get rid of the current limitation of having a single book
> with everything on a single directory. Sphinx has already hacks for it
> with "latex_documents", "man_pages", "texinfo_documents" conf.py variables
> that are specific for non-html builders.
>
> Still, when such variables are used, a post-sphinx-build logic with a
> per-output-file loop is needed.
>
>> I.e. even if we figured out how to
>> do intersphinx books, you'd want to grab parts from them and turn them
>> into something else?
>
> Either doing it or not, intersphinx is intestesting.
>
>> Ugh.
>>
>>
>> BR,
>> Jani.
>>
>>
>> --
>> Jani Nikula, Intel
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-15 14:33 ` Jani Nikula
@ 2025-09-15 15:05 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-15 15:05 UTC (permalink / raw)
To: Jani Nikula
Cc: Mauro Carvalho Chehab, Akira Yokosawa, corbet, linux-doc,
linux-kernel, tmgross
On Mon, Sep 15, 2025 at 05:33:37PM +0300, Jani Nikula wrote:
> On Mon, 15 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > On Mon, Sep 15, 2025 at 03:54:26PM +0300, Jani Nikula wrote:
> >> On Mon, 15 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >> > IMHO, long term solution is to change SPHINXDIRS into something
> >> > like:
> >> >
> >> > make O=doc_build SPHINXTITLE="Media docs" SPHINXDIRS="admin-guide/media userspace-api/media driver-api/media/"
> >> >
> >> > would create something similar to this(*):
> >> >
> >> > doc_build/sphindirs/
> >> > |
> >> > +--> index.rst
> >> > +--> admin-guide -> {srcdir}/Documentation/admin-guide/media/
> >> > +--> usespace-api -> {srcdir}/Documentation/admin-guide/media/
> >> > \--> driver-api -> {srcdir}/Documentation/admin-guide/media/
> >>
> >> So you're basically suggesting the documentation build should support
> >> cherry-picking parts of the documentation with categories different from
> >> what the upstream documentation has?
> >
> > No. I'm saying that, if we want to have a single build process
> > for multiple sphinxdirs, that sounds to be the better way to do it
> > to override sphinx-build limitation of having single source directory.
> >
> > The advantages is that:
> > - brings more performance, as a single build would be enough;
> > - cross-references between them will be properly solved.
> >
> > The disadvantages are:
> > - it would very likely need to create copies (or hard symlinks)
> > at the build dir, which may reduce performance;
> > - yet-another-hack;
> > - increased build complexity.
> >
> > I'm not convinced myself about doing it or not. I didn't like when
> > I had to do that after the media book was split on 3 books. If one thinks
> > that having for loops to build targets is a problem, we need a separate
> > discussion about how to avoid it. Also, this is outside of the scope of
> > this series.
>
> I honestly don't even understand what you're saying above
Perhaps it is due to the lack of context. I was replying some comments
from Akira where he mentioned about cherry-picking *.tex files after
sphinx build, and do some tricks to build all of them altogether.
My reply to his comments is that, if we're willing to cherry-pick things,
it is better/cleaner/safer to do it at the beginning, before even running
sphinx-build, ensuring no conflicts at the filename mapping.
Yet, analyzing the alternative I proposed, I see both pros and cons - up
to the point that I'm not convinced myself if it is worth doing such
changes upstream or not.
> and how it
> contradicts with what I said about cherry-picking the documentation to
> build.
It doesn't.
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-10 12:59 ` Mauro Carvalho Chehab
2025-09-10 13:33 ` Mauro Carvalho Chehab
@ 2025-09-11 10:23 ` Jani Nikula
2025-09-11 11:37 ` Mauro Carvalho Chehab
1 sibling, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-11 10:23 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Jonathan Corbet, Linux Doc Mailing List, Björn Roy Baron,
Alex Gaynor, Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross,
linux-kernel, rust-for-linux
On Wed, 10 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Wed, 10 Sep 2025 13:46:17 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
>> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> > There are too much magic inside docs Makefile to properly run
>> > sphinx-build. Create an ancillary script that contains all
>> > kernel-related sphinx-build call logic currently at Makefile.
>> >
>> > Such script is designed to work both as an standalone command
>> > and as part of a Makefile. As such, it properly handles POSIX
>> > jobserver used by GNU make.
>> >
>> > On a side note, there was a line number increase due to the
>> > conversion:
>> >
>> > Documentation/Makefile | 131 +++----------
>> > tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++
>> > 2 files changed, 323 insertions(+), 101 deletions(-)
>> >
>> > This is because some things are more verbosed on Python and because
>> > it requires reading env vars from Makefile. Besides it, this script
>> > has some extra features that don't exist at the Makefile:
>> >
>> > - It can be called directly from command line;
>> > - It properly return PDF build errors.
>> >
>> > When running the script alone, it will only take handle sphinx-build
>> > targets. On other words, it won't runn make rustdoc after building
>> > htmlfiles, nor it will run the extra check scripts.
>>
>> I've always strongly believed we should aim to make it possible to build
>> the documentation by running sphinx-build directly on the
>> command-line. Not that it would be the common way to run it, but to not
>> accumulate things in the Makefile that need to happen before or
>> after. To promote handling the documentation build in Sphinx. To be able
>> to debug issues and try new Sphinx versions without all the hacks.
>
> That would be the better, but, unfortunately, this is not possible, for
> several reasons:
>
> 1. SPHINXDIRS. It needs a lot of magic to work, both before running
> sphinx-build and after (inside conf.py);
Makes you wonder if that's the right solution to the original
problem. It was added as a kind of hack, and it stuck.
> 2. Several extensions require kernel-specific environment variables to
> work. Calling sphinx-build directly breaks them;
The extensions shouldn't be using environment variables for
configuration anyway. Add config options and set them in conf.py like
everything else?
> 3. Sphinx itself doesn't build several targets alone. Instead, they create
> a Makefile, and an extra step is needed to finish the build. That's
> the case for pdf and texinfo, for instance;
That's not true for the Makefile currently generated by
sphinx-quickstart. Granted, I haven't used Sphinx much for pdf output.
> 4. Man pages generation. Sphinx support to generate it is very poor;
In what way?
> 5. Rust integration adds more complexity to the table;
>
> I'm not seeing sphinx-build supporting the above needs anytime soon,
> and, even if we push our needs to Sphinx and it gets accepted there,
> we'll still need to wait for quite a while until LTS distros merge
> them.
I'm not suggesting to add anything to Sphinx upstream.
>> This patch moves a bunch of that logic into a Python wrapper, and I feel
>> like it complicates matters. You can no longer rely on 'make V=1' to get
>> the build commands, for instance.
>
> Quite the opposite. if you try using "make V=1", it won't show the
> command line used to call sphinx-build anymore.
>
> This series restore it.
>
> See, if you build with this series with V=1, you will see exactly
> what commands are used on the build:
>
> $ make V=1 htmldocs
> ...
> python3 ./tools/docs/sphinx-build-wrapper htmldocs \
> --sphinxdirs="." --conf="conf.py" \
> --builddir="Documentation/output" \
> --theme= --css= --paper=
> python3 /new_devel/docs/sphinx_latest/bin/sphinx-build -j25 -b html -c /new_devel/docs/Documentation -d /new_devel/docs/Documentation/output/.doctrees -D kerneldoc_bin=scripts/kernel-doc.py -D version=6.17.0-rc1 -D release=6.17.0-rc1+ -D kerneldoc_srctree=. /new_devel/docs/Documentation /new_devel/docs/Documentation/output
> ...
>
>
>
>> Newer Sphinx versions have the -M option for "make mode". The Makefiles
>> produced by sphinx-quickstart only have one build target:
>>
>> # Catch-all target: route all unknown targets to Sphinx using the new
>> # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
>
> I didn't know about this, but from [1] it sounds it covers just two
> targets: "latexpdf" and "info".
sphinx-build -M help gives a list of 24 targets.
> The most complex scenario is still not covered: SPHINXDIRS.
>
> [1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html
>
>> %: Makefile
>> @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
>>
>> That's all.
>
> Try doing such change on your makefile. it will break:
>
> - SPHINXDIRS;
> - V=1;
> - rustdoc
I know it does. That's the problem.
> and will still be dependent on variables that are passed via
> env from Kernel makefile. So, stil you can't run from command
> line. Also, if you call sphinx-build from command line:
>
> $ sphinx-build -j25 -b html Documentation Documentation/output
> ...
> File "<frozen os>", line 717, in __getitem__
> KeyError: 'srctree'
>
> It won't work, as several parameters that are required by conf.py and by
> Sphinx extensions would be missing (the most important one is srctree, but
> there are others in the line too).
>
>> The proposed wrapper duplicates loads of code that's supposed to be
>> handled by sphinx-build directly.
>
> Once we get the wrapper, we can work to simplify it, but still I
> can't see how to get rid of it.
I just don't understand the mentality of first adding something complex,
and then working to simplify it.
Don't make it a Rube Goldberg machine in the first place.
>> Including the target/builder names.
>
> True, but this was a design decision taken lots of years ago: instead
> of:
> make html
>
> we're using:
>
> make htmldocs
>
> This series doesn't change that: either makefile or the script need
> to tho the namespace conversion.
In the above Makefile snippet that conversion would be $(@:docs=)
The clean Makefile way of checking for having Sphinx and the required
versions of Python and dependencies etc. would be a .PHONY target that
just checks, and doesn't do *anything* else. It shouldn't be part of the
sphinx-build rules.
PHONY += check-versions
check-versions:
sphinx-pre-install --version-check
htmldocs: check-versions
...
Or something like that.
>> Seems to me the goal should be to figure out *generic* wrappers for
>> handling parallelism, not Sphinx aware/specific.
>>
>>
>> BR,
>> Jani.
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-11 10:23 ` Jani Nikula
@ 2025-09-11 11:37 ` Mauro Carvalho Chehab
2025-09-11 13:38 ` Jonathan Corbet
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-11 11:37 UTC (permalink / raw)
To: Jani Nikula
Cc: Mauro Carvalho Chehab, Jonathan Corbet, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
On Thu, Sep 11, 2025 at 01:23:55PM +0300, Jani Nikula wrote:
> On Wed, 10 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Em Wed, 10 Sep 2025 13:46:17 +0300
> > Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> >
> >> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >> > There are too much magic inside docs Makefile to properly run
> >> > sphinx-build. Create an ancillary script that contains all
> >> > kernel-related sphinx-build call logic currently at Makefile.
> >> >
> >> > Such script is designed to work both as an standalone command
> >> > and as part of a Makefile. As such, it properly handles POSIX
> >> > jobserver used by GNU make.
> >> >
> >> > On a side note, there was a line number increase due to the
> >> > conversion:
> >> >
> >> > Documentation/Makefile | 131 +++----------
> >> > tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++
> >> > 2 files changed, 323 insertions(+), 101 deletions(-)
> >> >
> >> > This is because some things are more verbosed on Python and because
> >> > it requires reading env vars from Makefile. Besides it, this script
> >> > has some extra features that don't exist at the Makefile:
> >> >
> >> > - It can be called directly from command line;
> >> > - It properly return PDF build errors.
> >> >
> >> > When running the script alone, it will only take handle sphinx-build
> >> > targets. On other words, it won't runn make rustdoc after building
> >> > htmlfiles, nor it will run the extra check scripts.
> >>
> >> I've always strongly believed we should aim to make it possible to build
> >> the documentation by running sphinx-build directly on the
> >> command-line. Not that it would be the common way to run it, but to not
> >> accumulate things in the Makefile that need to happen before or
> >> after. To promote handling the documentation build in Sphinx. To be able
> >> to debug issues and try new Sphinx versions without all the hacks.
> >
> > That would be the better, but, unfortunately, this is not possible, for
> > several reasons:
> >
> > 1. SPHINXDIRS. It needs a lot of magic to work, both before running
> > sphinx-build and after (inside conf.py);
>
> Makes you wonder if that's the right solution to the original
> problem. It was added as a kind of hack, and it stuck.
The problem is, IMHO, due to the lack of flexibility of sphinx-build:
It should have a way on it to do partial documentation builds.
On our specific case, given that building docs takes so much time,
SPHINXDIRS is needed, as it allows to quickly check how a change is
output.
I use it a lot at devel time.
Also, media documentation builds depend on it:
https://linuxtv.org/downloads/v4l-dvb-apis-new/
And probably other subsystems do the same, confining docs into
a subsystem-specific document.
>
> > 2. Several extensions require kernel-specific environment variables to
> > work. Calling sphinx-build directly breaks them;
>
> The extensions shouldn't be using environment variables for
> configuration anyway. Add config options and set them in conf.py like
> everything else?
Agreed, but that's a separate problem, and should be addressed outside
this path changeset.
This one give one step on such direction by passing some parameters
via -D instead of env, but still conf.py and scripts are handling
them on a very particular way.
> > 3. Sphinx itself doesn't build several targets alone. Instead, they create
> > a Makefile, and an extra step is needed to finish the build. That's
> > the case for pdf and texinfo, for instance;
>
> That's not true for the Makefile currently generated by
> sphinx-quickstart. Granted, I haven't used Sphinx much for pdf output.
At the beginning, we were relying on the auto-generated pdf makefiles
only, but this had several issues (I can't recall them anymore). So,
we ended to a more complex proccess.
Yet, still broken, as, no matter if using sphinx quickstart way
or not, one needs to run (directly or indirectly) the produced
Makefile inside Documentation/output/.../latex to generate files,
and it will still always return non-zero, even if all PDFs are
built.
The real fix for it is outside our hands: someone needs to change
the way PDF is produced with a proper PDF builder instead of latex
and integrate at Sphinx tree.
There are some OOT docutils and/or sphinx pdf builders. We tried for
a while rst2pdf, but never worked for the Kernel source.
>
> > 4. Man pages generation. Sphinx support to generate it is very poor;
>
> In what way?
Have you ever tried to use it?
"This builder produces manual pages in the groff format.
You have to specify which documents are to be included in
which manual pages via the man_pages configuration value."
(https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.manpage.ManualPageBuilder)
I tried when we were migrating to Sphinx. In summary:
- Each man page requires an entry on a list;
man_pages = [
('man/func', 'func', 'Func Documentation', [authors], 1),
...
]
- Each man page requires a .rst file;
- Each man page .rst file requires an specific format, like this:
func(9)
=======
NAME
----
func - ...
SYNOPSIS
--------
...
DESCRIPTION
-----------
...
E.g. it is basically a troff source "converted" to .rst.
> > 5. Rust integration adds more complexity to the table;
> >
> > I'm not seeing sphinx-build supporting the above needs anytime soon,
> > and, even if we push our needs to Sphinx and it gets accepted there,
> > we'll still need to wait for quite a while until LTS distros merge
> > them.
>
> I'm not suggesting to add anything to Sphinx upstream.
Without Sphinx upstream changes, I can't see how we'll get rid of
sphinx-build pre/post processing.
> >> This patch moves a bunch of that logic into a Python wrapper, and I feel
> >> like it complicates matters. You can no longer rely on 'make V=1' to get
> >> the build commands, for instance.
> >
> > Quite the opposite. if you try using "make V=1", it won't show the
> > command line used to call sphinx-build anymore.
> >
> > This series restore it.
> >
> > See, if you build with this series with V=1, you will see exactly
> > what commands are used on the build:
> >
> > $ make V=1 htmldocs
> > ...
> > python3 ./tools/docs/sphinx-build-wrapper htmldocs \
> > --sphinxdirs="." --conf="conf.py" \
> > --builddir="Documentation/output" \
> > --theme= --css= --paper=
> > python3 /new_devel/docs/sphinx_latest/bin/sphinx-build -j25 -b html -c /new_devel/docs/Documentation -d /new_devel/docs/Documentation/output/.doctrees -D kerneldoc_bin=scripts/kernel-doc.py -D version=6.17.0-rc1 -D release=6.17.0-rc1+ -D kerneldoc_srctree=. /new_devel/docs/Documentation /new_devel/docs/Documentation/output
> > ...
> >
> >
> >
> >> Newer Sphinx versions have the -M option for "make mode". The Makefiles
> >> produced by sphinx-quickstart only have one build target:
> >>
> >> # Catch-all target: route all unknown targets to Sphinx using the new
> >> # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
> >
> > I didn't know about this, but from [1] it sounds it covers just two
> > targets: "latexpdf" and "info".
>
> sphinx-build -M help gives a list of 24 targets.
Ok, but those are the extra ones. Btw, I'm almost sure we tried it for
latexpdf in the early days. Didn't work well. I guess the problem as
related to returned error codes that are always causing make pdfdocs
to return errors.
> > The most complex scenario is still not covered: SPHINXDIRS.
> >
> > [1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html
> >
> >> %: Makefile
> >> @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
> >>
> >> That's all.
> >
> > Try doing such change on your makefile. it will break:
> >
> > - SPHINXDIRS;
> > - V=1;
> > - rustdoc
>
> I know it does. That's the problem.
The wrapper fixes it by handling the pre/post sphinx-build logic
at the right way.
See, this series doesn't change the need or the implementation logic
for pre/post sphinx-build steps(*). It just moves them to a wrapper,
while getting rid of all small hacky scripts that are needed to make
them work.
We can and should still pursue the goal of simplifying pre/post
steps.
(*) The only change is for PDF: after the series, the return code
is now zero if all PDF files build, non-zero otherwise.
This makes easier for humans and for CI to be sure that there
are nothing at the .rst files causing PDF builds to break.
>
> > and will still be dependent on variables that are passed via
> > env from Kernel makefile. So, stil you can't run from command
> > line. Also, if you call sphinx-build from command line:
> >
> > $ sphinx-build -j25 -b html Documentation Documentation/output
> > ...
> > File "<frozen os>", line 717, in __getitem__
> > KeyError: 'srctree'
> >
> > It won't work, as several parameters that are required by conf.py and by
> > Sphinx extensions would be missing (the most important one is srctree, but
> > there are others in the line too).
> >
> >> The proposed wrapper duplicates loads of code that's supposed to be
> >> handled by sphinx-build directly.
> >
> > Once we get the wrapper, we can work to simplify it, but still I
> > can't see how to get rid of it.
>
> I just don't understand the mentality of first adding something complex,
> and then working to simplify it.
It doesn't make it more complex. Quite the opposite:
- it ports what we have as-is to a script;
- it drops hacky glues that were added over time on 4 different files
to handle sphinxdirs, jobserver, parallelism, second build steps,
man pages generation;
- it fixes the return code issue with pdf builds.
As all pre/post steps are now in a single place, it makes it easier to
maintain.
> Don't make it a Rube Goldberg machine in the first place.
>
> >> Including the target/builder names.
> >
> > True, but this was a design decision taken lots of years ago: instead
> > of:
> > make html
> >
> > we're using:
> >
> > make htmldocs
> >
> > This series doesn't change that: either makefile or the script need
> > to tho the namespace conversion.
>
> In the above Makefile snippet that conversion would be $(@:docs=)
The same could be done at python. Yet, I opted there to use a dict
mainly because:
- we're not consistent about where files will be stored:
with sphinx-quickstart, html files goes into _build/html. On our
build system, we dropped "/html";
- we have some specific rules about where the final PDF files will
be stored;
- we need to map what builder is used, because of the second step.
So, instead of having code checking for those specifics, I opted to
place on a dict, as it makes clearer and easier to maintain.
> The clean Makefile way of checking for having Sphinx and the required
> versions of Python and dependencies etc. would be a .PHONY target that
> just checks, and doesn't do *anything* else. It shouldn't be part of the
> sphinx-build rules.
>
> PHONY += check-versions
> check-versions:
> sphinx-pre-install --version-check
>
> htmldocs: check-versions
> ...
>
> Or something like that.
The problem with that is that we shouldn't run sphinx-pre-install for
cleandocs or non-doc targets.
Anyway, this series doesn't touch sphinx-pre-install call. It is still
inside docs Makefile.
What we can do in the future is to convert sphinx-pre-install code into
a library and then call a check_versions() method from it at the wrapper
script, like:
SphinxPreInstal().check_versions() # or something equivalent
And then dropping it from the build system. Yet, this is out of the scope
of this series.
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-11 11:37 ` Mauro Carvalho Chehab
@ 2025-09-11 13:38 ` Jonathan Corbet
2025-09-11 19:33 ` Jani Nikula
2025-09-12 8:28 ` Mauro Carvalho Chehab
0 siblings, 2 replies; 67+ messages in thread
From: Jonathan Corbet @ 2025-09-11 13:38 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jani Nikula
Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> On Thu, Sep 11, 2025 at 01:23:55PM +0300, Jani Nikula wrote:
>> > 1. SPHINXDIRS. It needs a lot of magic to work, both before running
>> > sphinx-build and after (inside conf.py);
>>
>> Makes you wonder if that's the right solution to the original
>> problem. It was added as a kind of hack, and it stuck.
>
> The problem is, IMHO, due to the lack of flexibility of sphinx-build:
> It should have a way on it to do partial documentation builds.
A couple of times I have looked into using intersphinx, making each book
into an actually separate book. The thing I always run into is that
doing a complete docs build, with working references, would require
building everything twice. This is probably worth another attempt one
of these years...
jon
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-11 13:38 ` Jonathan Corbet
@ 2025-09-11 19:33 ` Jani Nikula
2025-09-11 19:47 ` Jonathan Corbet
2025-09-12 8:28 ` Mauro Carvalho Chehab
1 sibling, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-11 19:33 UTC (permalink / raw)
To: Jonathan Corbet, Mauro Carvalho Chehab
Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
On Thu, 11 Sep 2025, Jonathan Corbet <corbet@lwn.net> wrote:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
>> On Thu, Sep 11, 2025 at 01:23:55PM +0300, Jani Nikula wrote:
>>> > 1. SPHINXDIRS. It needs a lot of magic to work, both before running
>>> > sphinx-build and after (inside conf.py);
>>>
>>> Makes you wonder if that's the right solution to the original
>>> problem. It was added as a kind of hack, and it stuck.
>>
>> The problem is, IMHO, due to the lack of flexibility of sphinx-build:
>> It should have a way on it to do partial documentation builds.
>
> A couple of times I have looked into using intersphinx, making each book
> into an actually separate book. The thing I always run into is that
> doing a complete docs build, with working references, would require
> building everything twice. This is probably worth another attempt one
> of these years...
I think the main factor in that should be whether it makes sense from
overall documentation standpoint, not the technical details.
Having several books might make sense. It might even be helpful in
organizing the documentation by audiences. But having the granularity of
SPHINXDIRS with that would be overkill. And there needs to be a book to
bring them together, and link to the other books, acting as the landing
page.
I believe it should be possible to generate the intersphinx inventory
without generating the full html or pdf documentation. So I don't think
it's actually two complete docs builds. It might speed things up to have
a number of independent documentation builds.
As to the working references, IIUC partial builds with SPHINXDIRS
doesn't get that part right if there are references outside of the
designated dirs, leading to warnings.
BR,
Jani.
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-11 19:33 ` Jani Nikula
@ 2025-09-11 19:47 ` Jonathan Corbet
2025-09-12 8:06 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jonathan Corbet @ 2025-09-11 19:47 UTC (permalink / raw)
To: Jani Nikula, Mauro Carvalho Chehab
Cc: Mauro Carvalho Chehab, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
Jani Nikula <jani.nikula@linux.intel.com> writes:
> On Thu, 11 Sep 2025, Jonathan Corbet <corbet@lwn.net> wrote:
>> A couple of times I have looked into using intersphinx, making each book
>> into an actually separate book. The thing I always run into is that
>> doing a complete docs build, with working references, would require
>> building everything twice. This is probably worth another attempt one
>> of these years...
>
> I think the main factor in that should be whether it makes sense from
> overall documentation standpoint, not the technical details.
>
> Having several books might make sense. It might even be helpful in
> organizing the documentation by audiences. But having the granularity of
> SPHINXDIRS with that would be overkill. And there needs to be a book to
> bring them together, and link to the other books, acting as the landing
> page.
Well, I think that the number of existing directories needs to be
reduced rather further. I made progress in that direction by coalescing
all the arch docs under Documentation/arch/. I would like to do
something similar with all the device-specific docs, creating
Documentation/devices/. Then we start to get to a reasonable number of
books.
> I believe it should be possible to generate the intersphinx inventory
> without generating the full html or pdf documentation. So I don't think
> it's actually two complete docs builds. It might speed things up to have
> a number of independent documentation builds.
That's a good point, I hadn't looked into that part. The builder phase
takes a lot of the time, if that could be cut out things would go
faster.
> As to the working references, IIUC partial builds with SPHINXDIRS
> doesn't get that part right if there are references outside of the
> designated dirs, leading to warnings.
That is true. My point though is that, to get the references right with
a *full* build, a two-pass approach is needed though, as you suggest,
perhaps the first pass could be faster.
Thanks,
jon
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-11 19:47 ` Jonathan Corbet
@ 2025-09-12 8:06 ` Mauro Carvalho Chehab
2025-09-12 10:16 ` Jani Nikula
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 8:06 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Jani Nikula, Linux Doc Mailing List, Björn Roy Baron,
Alex Gaynor, Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross,
linux-kernel, rust-for-linux
Em Thu, 11 Sep 2025 13:47:54 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Jani Nikula <jani.nikula@linux.intel.com> writes:
>
> > On Thu, 11 Sep 2025, Jonathan Corbet <corbet@lwn.net> wrote:
> >> A couple of times I have looked into using intersphinx, making each book
> >> into an actually separate book. The thing I always run into is that
> >> doing a complete docs build, with working references, would require
> >> building everything twice. This is probably worth another attempt one
> >> of these years...
There are a couple of different usecase scenarios for building docs.
1) The first and most important one is to produce book(s) for people
to use. This is usually done by some automation, and the result is
placed on places like:
- https://docs.kernel.org/
and on subsystem-specific places like:
- https://linuxtv.org/downloads/v4l-dvb-apis-new/
for scenario (1), taking twice the time to build is not an issue, as
nobody will be sitting on a chair waiting for the build to finish.
On such scenario, SPHINXDIRS is important on subsystem-specific docs.
For instance, on media, we use SPHINXDIRS to pick parts of 3 different
books:
- Documentation/admin-guide/media/
- Documentation/driver-api/media/
- Documentation/userspace-api/media/
What media automation does, once per day, is:
# Non-essencial parts of index.rst dropped
cat <<END >Documentation/media/index.rst
================================
Linux Kernel Media Documentation
================================
.. toctree::
admin-guide/index
driver-api/index
userspace-api/index
END
rsync -uAXEHlaSx -W --inplace --delete Documentation/admin-guide/media/ Documentation/media/admin-guide
rsync -uAXEHlaSx -W --inplace --delete Documentation/driver-api/media/ Documentation/media/driver-api
rsync -uAXEHlaSx -W --inplace --delete Documentation/userspace-api/media/ Documentation/media/userspace-api
make SPHINXDIRS='media' CSS='$CSS' DOCS_THEME='$DOCS_THEME' htmldocs
make SPHINXDIRS='media' pdfdocs
make SPHINXDIRS='media' epubdocs
2) CI tests. Here, taking more time usually is not a problem, except
when CI is used before pushing stuff, and the developer has to wait
it to finish before pushing.
For scenario (2), a build time increase is problematic, as, if it now
takes twice the time, a change like that will require twice the
resources for the build with may increase costs.
3) developers who touched docs. They want a way to quickly build and
verify the output for their changes.
Here, any time increase is problematic, and SPHINXDIRS play an important
hole by allowing them to build only the touched documents.
For instance, when I was developing Netlink yaml plugin, I had to use
dozens of times:
make SPINXDRS=Documentation/netlink/specs/ htmldocs
If I had to build the entire documentation every time, the development
time would increase from days to weeks.
Looking on those three scenarios, the only one where intersphinx is
useful is (1).
From my PoV, we should support intersphinx, but this should be optional.
Also, one has to point from where intersphinx will point unsolved
symbols. So, we would need something like:
make SPHINXREFMAP=intersphinx_mapping.py htmldocs
where intersphinx_mapping.py would be a file containing intersphinx
configuration. We would add a default map at Documentation/, while
letting it to be overridden if some subsystem has different requirements
or is using a different CSS tamplate or not using alabaster.
> > I think the main factor in that should be whether it makes sense from
> > overall documentation standpoint, not the technical details.
Agreed.
> > Having several books might make sense. It might even be helpful in
> > organizing the documentation by audiences. But having the granularity of
> > SPHINXDIRS with that would be overkill.
On the contrary. SPHINXDIRS granuarity is very important for scenario (3).
> > And there needs to be a book to
> > bring them together, and link to the other books, acting as the landing
> > page.
>
> Well, I think that the number of existing directories needs to be
> reduced rather further. I made progress in that direction by coalescing
> all the arch docs under Documentation/arch/. I would like to do
> something similar with all the device-specific docs, creating
> Documentation/devices/. Then we start to get to a reasonable number of
> books.
I don't think reducing the number of books should be the goal, but,
instead, to have them with a clear and coherent organization with focus
on the audience that will be actually using them.
After reorg, we may have less books. That's fine. But it is also fine
if we end with more books.
I lost the battle years ago, but I still believe that, at least for
some subsystems like media, i2c, DRM, security and others, a
subsystem-specific book could be better. After all, the audience for
such subsystems is very specialized.
> > I believe it should be possible to generate the intersphinx inventory
> > without generating the full html or pdf documentation. So I don't think
> > it's actually two complete docs builds. It might speed things up to have
> > a number of independent documentation builds.
>
> That's a good point, I hadn't looked into that part. The builder phase
> takes a lot of the time, if that could be cut out things would go
> faster.
Indeed, but we need to double check if .doctree cache expiration will
happen the right way for all books affected by a partial build.
During this merge window, I sent a RFC patch in the middle of a comment
with a conf.py logic to detect Sphinx cache expiration. I remember I
added a comment asking if we should upstream it or not, but, as nobody
answered, I ended forgetting about it.
If we're willing to experiment with that, I recommend looking on such
patch and add a variant of it, enabled via V=1 or via some debug
parameter.
The goal would be to check if a change on a file will ensure that all
books using it will have cache expiration and be rebuilt.
> > As to the working references, IIUC partial builds with SPHINXDIRS
> > doesn't get that part right if there are references outside of the
> > designated dirs, leading to warnings.
>
> That is true. My point though is that, to get the references right with
> a *full* build, a two-pass approach is needed though, as you suggest,
> perhaps the first pass could be faster.
How fast? during development time, SPHINXDIRS means a couple of seconds:
$ make clean; time make SPHINXDIRS="peci" htmldocs
...
real 0m1,373s
user 0m1,348s
Even more complex builds, even when picking more than one book, like this:
$ make clean; time make SPHINXDIRS="driver-api/media/ userspace-api/media/" htmldocs
...
real 0m11,801s
user 0m31,381s
sys 0m6,880s
it still fits at the seconds range. Can interphinx first pass have a
similar build time?
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 8:06 ` Mauro Carvalho Chehab
@ 2025-09-12 10:16 ` Jani Nikula
2025-09-12 11:34 ` Vegard Nossum
2025-09-12 11:41 ` Mauro Carvalho Chehab
0 siblings, 2 replies; 67+ messages in thread
From: Jani Nikula @ 2025-09-12 10:16 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jonathan Corbet
Cc: Linux Doc Mailing List, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross, linux-kernel,
rust-for-linux
On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Thu, 11 Sep 2025 13:47:54 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>> Jani Nikula <jani.nikula@linux.intel.com> writes:
>>
>> > On Thu, 11 Sep 2025, Jonathan Corbet <corbet@lwn.net> wrote:
>> >> A couple of times I have looked into using intersphinx, making each book
>> >> into an actually separate book. The thing I always run into is that
>> >> doing a complete docs build, with working references, would require
>> >> building everything twice. This is probably worth another attempt one
>> >> of these years...
>
> There are a couple of different usecase scenarios for building docs.
>
> 1) The first and most important one is to produce book(s) for people
> to use. This is usually done by some automation, and the result is
> placed on places like:
> - https://docs.kernel.org/
>
> and on subsystem-specific places like:
> - https://linuxtv.org/downloads/v4l-dvb-apis-new/
>
> for scenario (1), taking twice the time to build is not an issue, as
> nobody will be sitting on a chair waiting for the build to finish.
>
> On such scenario, SPHINXDIRS is important on subsystem-specific docs.
> For instance, on media, we use SPHINXDIRS to pick parts of 3 different
> books:
>
> - Documentation/admin-guide/media/
> - Documentation/driver-api/media/
> - Documentation/userspace-api/media/
>
> What media automation does, once per day, is:
>
> # Non-essencial parts of index.rst dropped
> cat <<END >Documentation/media/index.rst
> ================================
> Linux Kernel Media Documentation
> ================================
>
> .. toctree::
>
> admin-guide/index
> driver-api/index
> userspace-api/index
> END
>
> rsync -uAXEHlaSx -W --inplace --delete Documentation/admin-guide/media/ Documentation/media/admin-guide
> rsync -uAXEHlaSx -W --inplace --delete Documentation/driver-api/media/ Documentation/media/driver-api
> rsync -uAXEHlaSx -W --inplace --delete Documentation/userspace-api/media/ Documentation/media/userspace-api
>
> make SPHINXDIRS='media' CSS='$CSS' DOCS_THEME='$DOCS_THEME' htmldocs
> make SPHINXDIRS='media' pdfdocs
> make SPHINXDIRS='media' epubdocs
I was actually wondering how [1] was built. So it's not a complete build
of anything upstream, but rather something cobbled together downstream.
So your scenario (1) above is actually *two* wildly different scenarios.
And if upstream needs to cater for pretty much random subsets of
documentation being built, cherry-picking documentation from here and
there, I don't know what hope there is in radically refactoring how
documentation gets built upstream.
I presume you have one or more of a) get bunch of broken link warnings
at build, b) get broken links in the output, c) avoid links outside of
your subset altogether.
[1] https://linuxtv.org/downloads/v4l-dvb-apis-new/
> 2) CI tests. Here, taking more time usually is not a problem, except
> when CI is used before pushing stuff, and the developer has to wait
> it to finish before pushing.
>
> For scenario (2), a build time increase is problematic, as, if it now
> takes twice the time, a change like that will require twice the
> resources for the build with may increase costs.
>
> 3) developers who touched docs. They want a way to quickly build and
> verify the output for their changes.
>
> Here, any time increase is problematic, and SPHINXDIRS play an important
> hole by allowing them to build only the touched documents.
This is actually problematic, because the SPHINXDIRS partial builds will
give you warnings for unresolved references that are just fine if the
entire documentation gets built.
> For instance, when I was developing Netlink yaml plugin, I had to use
> dozens of times:
>
> make SPINXDRS=Documentation/netlink/specs/ htmldocs
>
> If I had to build the entire documentation every time, the development
> time would increase from days to weeks.
>
> Looking on those three scenarios, the only one where intersphinx is
> useful is (1).
It's also helpful for 3, and it could be helpful for 2 if CI only checks
some parts of the documentation.
> From my PoV, we should support intersphinx, but this should be optional.
Per my understanding making this somehow optional is not easily
achieved. And you end up with a bunch of extra complexity.
> Also, one has to point from where intersphinx will point unsolved
> symbols. So, we would need something like:
>
> make SPHINXREFMAP=intersphinx_mapping.py htmldocs
>
> where intersphinx_mapping.py would be a file containing intersphinx
> configuration. We would add a default map at Documentation/, while
> letting it to be overridden if some subsystem has different requirements
> or is using a different CSS tamplate or not using alabaster.
>
>> > I think the main factor in that should be whether it makes sense from
>> > overall documentation standpoint, not the technical details.
>
> Agreed.
>
>> > Having several books might make sense. It might even be helpful in
>> > organizing the documentation by audiences. But having the granularity of
>> > SPHINXDIRS with that would be overkill.
>
> On the contrary. SPHINXDIRS granuarity is very important for scenario (3).
Sphinx does support incremental builds, and it's only the very first
build that's slow. IMO a handful of books that you can actually build
without warnings (unlike SPHINXDIRS) with incremental builds is a good
compromise.
>> > And there needs to be a book to
>> > bring them together, and link to the other books, acting as the landing
>> > page.
>>
>> Well, I think that the number of existing directories needs to be
>> reduced rather further. I made progress in that direction by coalescing
>> all the arch docs under Documentation/arch/. I would like to do
>> something similar with all the device-specific docs, creating
>> Documentation/devices/. Then we start to get to a reasonable number of
>> books.
>
> I don't think reducing the number of books should be the goal, but,
> instead, to have them with a clear and coherent organization with focus
> on the audience that will be actually using them.
>
> After reorg, we may have less books. That's fine. But it is also fine
> if we end with more books.
>
> I lost the battle years ago, but I still believe that, at least for
> some subsystems like media, i2c, DRM, security and others, a
> subsystem-specific book could be better. After all, the audience for
> such subsystems is very specialized.
>
>> > I believe it should be possible to generate the intersphinx inventory
>> > without generating the full html or pdf documentation. So I don't think
>> > it's actually two complete docs builds. It might speed things up to have
>> > a number of independent documentation builds.
>>
>> That's a good point, I hadn't looked into that part. The builder phase
>> takes a lot of the time, if that could be cut out things would go
>> faster.
>
> Indeed, but we need to double check if .doctree cache expiration will
> happen the right way for all books affected by a partial build.
>
> During this merge window, I sent a RFC patch in the middle of a comment
> with a conf.py logic to detect Sphinx cache expiration. I remember I
> added a comment asking if we should upstream it or not, but, as nobody
> answered, I ended forgetting about it.
>
> If we're willing to experiment with that, I recommend looking on such
> patch and add a variant of it, enabled via V=1 or via some debug
> parameter.
>
> The goal would be to check if a change on a file will ensure that all
> books using it will have cache expiration and be rebuilt.
>
>> > As to the working references, IIUC partial builds with SPHINXDIRS
>> > doesn't get that part right if there are references outside of the
>> > designated dirs, leading to warnings.
>>
>> That is true. My point though is that, to get the references right with
>> a *full* build, a two-pass approach is needed though, as you suggest,
>> perhaps the first pass could be faster.
>
> How fast? during development time, SPHINXDIRS means a couple of seconds:
>
> $ make clean; time make SPHINXDIRS="peci" htmldocs
> ...
> real 0m1,373s
> user 0m1,348s
>
> Even more complex builds, even when picking more than one book, like this:
>
> $ make clean; time make SPHINXDIRS="driver-api/media/ userspace-api/media/" htmldocs
> ...
> real 0m11,801s
> user 0m31,381s
> sys 0m6,880s
>
> it still fits at the seconds range. Can interphinx first pass have a
> similar build time?
Probably not. Can you add links from media to non-media documentation
without warnings? Probably not also.
BR,
Jani.
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 10:16 ` Jani Nikula
@ 2025-09-12 11:34 ` Vegard Nossum
2025-09-13 10:18 ` Mauro Carvalho Chehab
2025-09-12 11:41 ` Mauro Carvalho Chehab
1 sibling, 1 reply; 67+ messages in thread
From: Vegard Nossum @ 2025-09-12 11:34 UTC (permalink / raw)
To: Jani Nikula, Mauro Carvalho Chehab, Jonathan Corbet
Cc: Linux Doc Mailing List, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross, linux-kernel,
rust-for-linux
On 12/09/2025 12:16, Jani Nikula wrote:
>> Here, any time increase is problematic, and SPHINXDIRS play an important
>> hole by allowing them to build only the touched documents.
> This is actually problematic, because the SPHINXDIRS partial builds will
> give you warnings for unresolved references that are just fine if the
> entire documentation gets built.
I admit I don't have a full overview of all the problems that are being
solved here (in existing and proposed code), but how hard would it be to
convert the whole SPHINXDIRS thing into a Sphinx plugin that runs early
and discards documents outside of what the user wants to build? By
"discards" I mean in some useful way that reduces runtime compared to a
full build while retaining some benefits of a full build (reference
checking)?
Vegard
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 11:34 ` Vegard Nossum
@ 2025-09-13 10:18 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-13 10:18 UTC (permalink / raw)
To: Vegard Nossum
Cc: Jani Nikula, Jonathan Corbet, Linux Doc Mailing List,
Björn Roy Baron, Alex Gaynor, Alice Ryhl, Boqun Feng,
Gary Guo, Trevor Gross, linux-kernel, rust-for-linux
Em Fri, 12 Sep 2025 13:34:17 +0200
Vegard Nossum <vegard.nossum@oracle.com> escreveu:
> On 12/09/2025 12:16, Jani Nikula wrote:
> >> Here, any time increase is problematic, and SPHINXDIRS play an important
> >> hole by allowing them to build only the touched documents.
> > This is actually problematic, because the SPHINXDIRS partial builds will
> > give you warnings for unresolved references that are just fine if the
> > entire documentation gets built.
>
> I admit I don't have a full overview of all the problems that are being
> solved here (in existing and proposed code), but how hard would it be to
> convert the whole SPHINXDIRS thing into a Sphinx plugin that runs early
> and discards documents outside of what the user wants to build? By
> "discards" I mean in some useful way that reduces runtime compared to a
> full build while retaining some benefits of a full build (reference
> checking)?
That's not a bad idea, but I guess it is not too easy to implement - at
least inside a Sphinx plugin.
The good news is that conf.py has already a logic to ignore patterns that
could be tweaked and/or placed on a plugin.
The bad news is that existing index.rst files will now reference
non-existing docs. No idea how to "process" them to filter out
such docs. It is probably doable. See, SPHINXDIRS supports pinpointing
any directory, not just Documentation child directories. So, eventually,
such plugin would also need to "fake" the main index.rst. Now, the
question is, if we pick for instance:
SPHINXDIRS="netlink/spec networking"
What would be the main title that would be at the main index.rst?
I suspect that, for such cases, the title of the index would need
to be manually set at the command line interface.
Another aspect is that SPHINXDIRS affect latex document lists,
which can be problematic.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-12 10:16 ` Jani Nikula
2025-09-12 11:34 ` Vegard Nossum
@ 2025-09-12 11:41 ` Mauro Carvalho Chehab
1 sibling, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 11:41 UTC (permalink / raw)
To: Jani Nikula
Cc: Jonathan Corbet, Linux Doc Mailing List, Björn Roy Baron,
Alex Gaynor, Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross,
linux-kernel, rust-for-linux
Em Fri, 12 Sep 2025 13:16:12 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Em Thu, 11 Sep 2025 13:47:54 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >
> >> Jani Nikula <jani.nikula@linux.intel.com> writes:
> >>
> >> > On Thu, 11 Sep 2025, Jonathan Corbet <corbet@lwn.net> wrote:
> >> >> A couple of times I have looked into using intersphinx, making each book
> >> >> into an actually separate book. The thing I always run into is that
> >> >> doing a complete docs build, with working references, would require
> >> >> building everything twice. This is probably worth another attempt one
> >> >> of these years...
> >
> > There are a couple of different usecase scenarios for building docs.
> >
> > 1) The first and most important one is to produce book(s) for people
> > to use. This is usually done by some automation, and the result is
> > placed on places like:
> > - https://docs.kernel.org/
> >
> > and on subsystem-specific places like:
> > - https://linuxtv.org/downloads/v4l-dvb-apis-new/
> >
> > for scenario (1), taking twice the time to build is not an issue, as
> > nobody will be sitting on a chair waiting for the build to finish.
> >
> > On such scenario, SPHINXDIRS is important on subsystem-specific docs.
> > For instance, on media, we use SPHINXDIRS to pick parts of 3 different
> > books:
> >
> > - Documentation/admin-guide/media/
> > - Documentation/driver-api/media/
> > - Documentation/userspace-api/media/
> >
> > What media automation does, once per day, is:
> >
> > # Non-essencial parts of index.rst dropped
> > cat <<END >Documentation/media/index.rst
> > ================================
> > Linux Kernel Media Documentation
> > ================================
> >
> > .. toctree::
> >
> > admin-guide/index
> > driver-api/index
> > userspace-api/index
> > END
> >
> > rsync -uAXEHlaSx -W --inplace --delete Documentation/admin-guide/media/ Documentation/media/admin-guide
> > rsync -uAXEHlaSx -W --inplace --delete Documentation/driver-api/media/ Documentation/media/driver-api
> > rsync -uAXEHlaSx -W --inplace --delete Documentation/userspace-api/media/ Documentation/media/userspace-api
> >
> > make SPHINXDIRS='media' CSS='$CSS' DOCS_THEME='$DOCS_THEME' htmldocs
> > make SPHINXDIRS='media' pdfdocs
> > make SPHINXDIRS='media' epubdocs
>
> I was actually wondering how [1] was built. So it's not a complete build
> of anything upstream, but rather something cobbled together downstream.
It used to be a direct build from upstream. I had to do this hack when
we decided to split subsystem docs on 3 separate books.
> So your scenario (1) above is actually *two* wildly different scenarios.
>
> And if upstream needs to cater for pretty much random subsets of
> documentation being built, cherry-picking documentation from here and
> there, I don't know what hope there is in radically refactoring how
> documentation gets built upstream.
>
> I presume you have one or more of a) get bunch of broken link warnings
> at build, b) get broken links in the output, c) avoid links outside of
> your subset altogether.
There aren't many broken links, and this is not due to (c): almost all
cross-references we have are between media kAPI and media uAPI. Those
were solved when we artificially joined two books and used SPHINXDIRS
feature to produce the docs.
If we had intersphinx support, I would be building the docs in
separate using the standard SPHINXDIRS logic to create such
cross references, pointing to linuxtv.org for media docs and to
docs.kernel.org for other ones.
>
> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/
>
> > 2) CI tests. Here, taking more time usually is not a problem, except
> > when CI is used before pushing stuff, and the developer has to wait
> > it to finish before pushing.
> >
> > For scenario (2), a build time increase is problematic, as, if it now
> > takes twice the time, a change like that will require twice the
> > resources for the build with may increase costs.
> >
> > 3) developers who touched docs. They want a way to quickly build and
> > verify the output for their changes.
> >
> > Here, any time increase is problematic, and SPHINXDIRS play an important
> > hole by allowing them to build only the touched documents.
>
> This is actually problematic, because the SPHINXDIRS partial builds will
> give you warnings for unresolved references that are just fine if the
> entire documentation gets built.
True, but if you pick them before/after a chanseset, you can notice
if the warning was introduced or not by the changeset. Only if it was
introduced by the patchset you need to wait 3 minutes for the full build.
> > For instance, when I was developing Netlink yaml plugin, I had to use
> > dozens of times:
> >
> > make SPINXDRS=Documentation/netlink/specs/ htmldocs
> >
> > If I had to build the entire documentation every time, the development
> > time would increase from days to weeks.
> >
> > Looking on those three scenarios, the only one where intersphinx is
> > useful is (1).
>
> It's also helpful for 3, and it could be helpful for 2 if CI only checks
> some parts of the documentation.
I'm not arguing against intersphinx. I do think having it is something
we need to aim for.
The question is: does it replace SPHINXDIRS by providing quick builds
if only some of the books were changed?
> > From my PoV, we should support intersphinx, but this should be optional.
>
> Per my understanding making this somehow optional is not easily
> achieved. And you end up with a bunch of extra complexity.
True, but I guess extra complexity is unavoidable: intersphinx
requires a list of books with reference locations, with is not the
same for everyone.
This is what expect once we have intersphinx in place:
Use linuxtv.org URLs for all references from:
- Documentation/admin-guide/media/
- Documentation/driver-api/media/
- Documentation/userspace-api/media/
everything else: from kernel.org.
As they were generated from media next branch.
If implement it for DRM, in a way to track what DRM next branches
have, and if you have kapi, uapi and per-driver apis on different
books, you will probably want to solve intersphinx dependencies with
a FDO specific "search" order, like:
- xe and i915 books: from intel next branches;
- amd books: from amd next branches;
- drm core: from drm-next;
- everything else: from kernel.org.
So, it is not just making it optional: you also need to provide a
way to allow it to be adjusted were it is needed. IMO, the easiest
way would be to have a separate .py file with intersphinx specifics:
make SPHINXREFMAP=intersphinx_mapping.py htmldocs
This way, I could create a media_mapping.py file that would include
intersphinx_mapping.py and replace some defaults to do my own mapping.
> > Also, one has to point from where intersphinx will point unsolved
> > symbols. So, we would need something like:
> >
> > make SPHINXREFMAP=intersphinx_mapping.py htmldocs
> >
> > where intersphinx_mapping.py would be a file containing intersphinx
> > configuration. We would add a default map at Documentation/, while
> > letting it to be overridden if some subsystem has different requirements
> > or is using a different CSS tamplate or not using alabaster.
> >
> >> > I think the main factor in that should be whether it makes sense from
> >> > overall documentation standpoint, not the technical details.
> >
> > Agreed.
> >
> >> > Having several books might make sense. It might even be helpful in
> >> > organizing the documentation by audiences. But having the granularity of
> >> > SPHINXDIRS with that would be overkill.
> >
> > On the contrary. SPHINXDIRS granuarity is very important for scenario (3).
>
> Sphinx does support incremental builds, and it's only the very first
> build that's slow. IMO a handful of books that you can actually build
> without warnings (unlike SPHINXDIRS) with incremental builds is a good
> compromise.
That's not quite true: when Sphinx detects a missing file, it expires
the caches related to it and don't do incremental builds anymore. I had
to write a patch during the last development cycle due to that, as -rc1
came up with a broken reference because of a file rename. This was only
solved 3 months after the fact.
> >> > And there needs to be a book to
> >> > bring them together, and link to the other books, acting as the landing
> >> > page.
> >>
> >> Well, I think that the number of existing directories needs to be
> >> reduced rather further. I made progress in that direction by coalescing
> >> all the arch docs under Documentation/arch/. I would like to do
> >> something similar with all the device-specific docs, creating
> >> Documentation/devices/. Then we start to get to a reasonable number of
> >> books.
> >
> > I don't think reducing the number of books should be the goal, but,
> > instead, to have them with a clear and coherent organization with focus
> > on the audience that will be actually using them.
> >
> > After reorg, we may have less books. That's fine. But it is also fine
> > if we end with more books.
> >
> > I lost the battle years ago, but I still believe that, at least for
> > some subsystems like media, i2c, DRM, security and others, a
> > subsystem-specific book could be better. After all, the audience for
> > such subsystems is very specialized.
> >
> >> > I believe it should be possible to generate the intersphinx inventory
> >> > without generating the full html or pdf documentation. So I don't think
> >> > it's actually two complete docs builds. It might speed things up to have
> >> > a number of independent documentation builds.
> >>
> >> That's a good point, I hadn't looked into that part. The builder phase
> >> takes a lot of the time, if that could be cut out things would go
> >> faster.
> >
> > Indeed, but we need to double check if .doctree cache expiration will
> > happen the right way for all books affected by a partial build.
> >
> > During this merge window, I sent a RFC patch in the middle of a comment
> > with a conf.py logic to detect Sphinx cache expiration. I remember I
> > added a comment asking if we should upstream it or not, but, as nobody
> > answered, I ended forgetting about it.
> >
> > If we're willing to experiment with that, I recommend looking on such
> > patch and add a variant of it, enabled via V=1 or via some debug
> > parameter.
> >
> > The goal would be to check if a change on a file will ensure that all
> > books using it will have cache expiration and be rebuilt.
> >
> >> > As to the working references, IIUC partial builds with SPHINXDIRS
> >> > doesn't get that part right if there are references outside of the
> >> > designated dirs, leading to warnings.
> >>
> >> That is true. My point though is that, to get the references right with
> >> a *full* build, a two-pass approach is needed though, as you suggest,
> >> perhaps the first pass could be faster.
> >
> > How fast? during development time, SPHINXDIRS means a couple of seconds:
> >
> > $ make clean; time make SPHINXDIRS="peci" htmldocs
> > ...
> > real 0m1,373s
> > user 0m1,348s
> >
> > Even more complex builds, even when picking more than one book, like this:
> >
> > $ make clean; time make SPHINXDIRS="driver-api/media/ userspace-api/media/" htmldocs
> > ...
> > real 0m11,801s
> > user 0m31,381s
> > sys 0m6,880s
> >
> > it still fits at the seconds range. Can interphinx first pass have a
> > similar build time?
>
> Probably not. Can you add links from media to non-media documentation
> without warnings? Probably not also.
No, but I can count on my fingers the number of times I made such
changes: 99.9% of the time, doc changes aren't on the few docs that
have subsystem interdependencies. Even the number of dependencies
between media kapi and uapi are not high.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-09-11 13:38 ` Jonathan Corbet
2025-09-11 19:33 ` Jani Nikula
@ 2025-09-12 8:28 ` Mauro Carvalho Chehab
1 sibling, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 8:28 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Jani Nikula, Linux Doc Mailing List, Björn Roy Baron,
Alex Gaynor, Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross,
linux-kernel, rust-for-linux
Em Thu, 11 Sep 2025 07:38:56 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
> > On Thu, Sep 11, 2025 at 01:23:55PM +0300, Jani Nikula wrote:
> >> > 1. SPHINXDIRS. It needs a lot of magic to work, both before running
> >> > sphinx-build and after (inside conf.py);
> >>
> >> Makes you wonder if that's the right solution to the original
> >> problem. It was added as a kind of hack, and it stuck.
> >
> > The problem is, IMHO, due to the lack of flexibility of sphinx-build:
> > It should have a way on it to do partial documentation builds.
>
> A couple of times I have looked into using intersphinx, making each book
> into an actually separate book. The thing I always run into is that
> doing a complete docs build, with working references, would require
> building everything twice. This is probably worth another attempt one
> of these years...
The big advantage of intersphinx is for PDF and LaTeX output, as
this is the only way to have cross-references there.
It is also good for subsystem-specific books (or "sub-"books) like:
- Documentation/admin-guide/media/
- Documentation/driver-api/media/
- Documentation/userspace-api/media/
Right now, we create a single book with all those tree, but I would
prefer to build each of them as separate units, as they are for separated
audiences, but only if cross-references will be solved in a way that
html and pdf docs will point to the other books stored at linuxtv.org.
For html, this won't be any different, in practice, from what we have,
but for PDF and ePub, this would mean smaller books.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* [PATCH v4 09/19] tools/docs: sphinx-build-wrapper: add comments and blank lines
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (7 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv Mauro Carvalho Chehab
` (10 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Andreas Hindborg, Benno Lossin, Boqun Feng,
Danilo Krummrich, Gary Guo, Mauro Carvalho Chehab, Miguel Ojeda,
Trevor Gross, linux-kernel, rust-for-linux
To help seing the actual size of the script when it was added,
I opted to strip out all comments from the original script.
Add them here:
tools/docs/sphinx-build-wrapper | 261 +++++++++++++++++++++++++++++++-
1 file changed, 257 insertions(+), 4 deletion(-)
As the code from the script has 288 lines of code, it means that
about half of the script are comments.
Also ensure pylint won't report any warnings.
No functional changes.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/sphinx-build-wrapper | 262 +++++++++++++++++++++++++++++++-
1 file changed, 258 insertions(+), 4 deletions(-)
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index 3256418d8dc5..ea9f8e17b0bc 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -1,21 +1,71 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2025 Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
+#
+# pylint: disable=R0902, R0912, R0913, R0914, R0915, R0917, C0103
+#
+# Converted from docs Makefile and parallel-wrapper.sh, both under
+# GPLv2, copyrighted since 2008 by the following authors:
+#
+# Akira Yokosawa <akiyks@gmail.com>
+# Arnd Bergmann <arnd@arndb.de>
+# Breno Leitao <leitao@debian.org>
+# Carlos Bilbao <carlos.bilbao@amd.com>
+# Dave Young <dyoung@redhat.com>
+# Donald Hunter <donald.hunter@gmail.com>
+# Geert Uytterhoeven <geert+renesas@glider.be>
+# Jani Nikula <jani.nikula@intel.com>
+# Jan Stancek <jstancek@redhat.com>
+# Jonathan Corbet <corbet@lwn.net>
+# Joshua Clayton <stillcompiling@gmail.com>
+# Kees Cook <keescook@chromium.org>
+# Linus Torvalds <torvalds@linux-foundation.org>
+# Magnus Damm <damm+renesas@opensource.se>
+# Masahiro Yamada <masahiroy@kernel.org>
+# Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
+# Maxim Cournoyer <maxim.cournoyer@gmail.com>
+# Peter Foley <pefoley2@pefoley.com>
+# Randy Dunlap <rdunlap@infradead.org>
+# Rob Herring <robh@kernel.org>
+# Shuah Khan <shuahkh@osg.samsung.com>
+# Thorsten Blum <thorsten.blum@toblux.com>
+# Tomas Winkler <tomas.winkler@intel.com>
+
+
+"""
+Sphinx build wrapper that handles Kernel-specific business rules:
+
+- it gets the Kernel build environment vars;
+- it determines what's the best parallelism;
+- it handles SPHINXDIRS
+
+This tool ensures that MIN_PYTHON_VERSION is satisfied. If version is
+below that, it seeks for a new Python version. If found, it re-runs using
+the newer version.
+"""
+
import argparse
import os
import shlex
import shutil
import subprocess
import sys
+
from lib.python_version import PythonVersion
LIB_DIR = "../../scripts/lib"
SRC_DIR = os.path.dirname(os.path.realpath(__file__))
+
sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
-from jobserver import JobserverExec
+from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
+#
+# Some constants
+#
MIN_PYTHON_VERSION = PythonVersion("3.7").version
PAPER = ["", "a4", "letter"]
+
TARGETS = {
"cleandocs": { "builder": "clean" },
"linkcheckdocs": { "builder": "linkcheck" },
@@ -28,8 +78,19 @@ TARGETS = {
"xmldocs": { "builder": "xml", "out_dir": "xml" },
}
+
+#
+# SphinxBuilder class
+#
+
class SphinxBuilder:
+ """
+ Handles a sphinx-build target, adding needed arguments to build
+ with the Kernel.
+ """
+
def is_rust_enabled(self):
+ """Check if rust is enabled at .config"""
config_path = os.path.join(self.srctree, ".config")
if os.path.isfile(config_path):
with open(config_path, "r", encoding="utf-8") as f:
@@ -37,41 +98,88 @@ class SphinxBuilder:
return False
def get_path(self, path, use_cwd=False, abs_path=False):
+ """
+ Ancillary routine to handle patches the right way, as shell does.
+
+ It first expands "~" and "~user". Then, if patch is not absolute,
+ join self.srctree. Finally, if requested, convert to abspath.
+ """
+
path = os.path.expanduser(path)
if not path.startswith("/"):
if use_cwd:
base = os.getcwd()
else:
base = self.srctree
+
path = os.path.join(base, path)
+
if abs_path:
return os.path.abspath(path)
+
return path
def __init__(self, builddir, verbose=False, n_jobs=None):
+ """Initialize internal variables"""
self.verbose = None
+
+ #
+ # Normal variables passed from Kernel's makefile
+ #
self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+
if not verbose:
verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
+
if verbose is not None:
self.verbose = verbose
+
+ #
+ # As we handle number of jobs and quiet in separate, we need to pick
+ # both the same way as sphinx-build would pick, optionally accepts
+ # whitespaces or not. So let's use argparse to handle argument expansion
+ #
parser = argparse.ArgumentParser()
parser.add_argument('-j', '--jobs', type=int)
parser.add_argument('-q', '--quiet', type=int)
+
+ #
+ # Other sphinx-build arguments go as-is, so place them
+ # at self.sphinxopts, using shell parser
+ #
sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", ""))
+
+ #
+ # Build a list of sphinx args
+ #
sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
if sphinx_args.quiet is True:
self.verbose = False
+
if sphinx_args.jobs:
self.n_jobs = sphinx_args.jobs
+
+ #
+ # If the command line argument "-j" is used override SPHINXOPTS
+ #
+
self.n_jobs = n_jobs
+
+ #
+ # Source tree directory. This needs to be at os.environ, as
+ # Sphinx extensions use it
+ #
self.srctree = os.environ.get("srctree")
if not self.srctree:
self.srctree = "."
os.environ["srctree"] = self.srctree
+
+ #
+ # Now that we can expand srctree, get other directories as well
+ #
self.sphinxbuild = os.environ.get("SPHINXBUILD", "sphinx-build")
self.kerneldoc = self.get_path(os.environ.get("KERNELDOC",
"scripts/kernel-doc.py"))
@@ -79,20 +187,36 @@ class SphinxBuilder:
self.config_rust = self.is_rust_enabled()
+ #
+ # Get directory locations for LaTeX build toolchain
+ #
self.pdflatex_cmd = shutil.which(self.pdflatex)
self.latexmk_cmd = shutil.which("latexmk")
self.env = os.environ.copy()
def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
+ """
+ Executes sphinx-build using current python3 command and setting
+ -j parameter if possible to run the build in parallel.
+ """
+
with JobserverExec() as jobserver:
if jobserver.claim:
n_jobs = str(jobserver.claim)
else:
n_jobs = "auto" # Supported since Sphinx 1.7
+
cmd = []
+
cmd.append(sys.executable)
+
cmd.append(sphinx_build)
+
+ #
+ # Override auto setting, if explicitly passed from command line
+ # or via SPHINXOPTS
+ #
if self.n_jobs:
n_jobs = str(self.n_jobs)
@@ -101,59 +225,100 @@ class SphinxBuilder:
if not self.verbose:
cmd.append("-q")
+
cmd += self.sphinxopts
cmd += build_args
+
if self.verbose:
print(" ".join(cmd))
return subprocess.call(cmd, *args, **pwargs)
def handle_html(self, css, output_dir):
+ """
+ Extra steps for HTML and epub output.
+
+ For such targets, we need to ensure that CSS will be properly
+ copied to the output _static directory
+ """
+
if not css:
return
+
css = os.path.expanduser(css)
if not css.startswith("/"):
css = os.path.join(self.srctree, css)
+
static_dir = os.path.join(output_dir, "_static")
os.makedirs(static_dir, exist_ok=True)
+
try:
shutil.copy2(css, static_dir)
except (OSError, IOError) as e:
print(f"Warning: Failed to copy CSS: {e}", file=sys.stderr)
def handle_pdf(self, output_dirs):
+ """
+ Extra steps for PDF output.
+
+ As PDF is handled via a LaTeX output, after building the .tex file,
+ a new build is needed to create the PDF output from the latex
+ directory.
+ """
builds = {}
max_len = 0
+
for from_dir in output_dirs:
pdf_dir = os.path.join(from_dir, "../pdf")
os.makedirs(pdf_dir, exist_ok=True)
+
if self.latexmk_cmd:
latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
else:
latex_cmd = [self.pdflatex]
+
latex_cmd.extend(shlex.split(self.latexopts))
+
tex_suffix = ".tex"
+
+ #
+ # Process each .tex file
+ #
+
has_tex = False
build_failed = False
with os.scandir(from_dir) as it:
for entry in it:
if not entry.name.endswith(tex_suffix):
continue
+
name = entry.name[:-len(tex_suffix)]
has_tex = True
+
+ #
+ # LaTeX PDF error code is almost useless for us:
+ # any warning makes it non-zero. For kernel doc builds it
+ # always return non-zero even when build succeeds.
+ # So, let's do the best next thing: check if all PDF
+ # files were built. If they're, print a summary and
+ # return 0 at the end of this function
+ #
try:
subprocess.run(latex_cmd + [entry.path],
cwd=from_dir, check=True)
except subprocess.CalledProcessError:
pass
+
pdf_name = name + ".pdf"
pdf_from = os.path.join(from_dir, pdf_name)
pdf_to = os.path.join(pdf_dir, pdf_name)
+
if os.path.exists(pdf_from):
os.rename(pdf_from, pdf_to)
builds[name] = os.path.relpath(pdf_to, self.builddir)
else:
builds[name] = "FAILED"
build_failed = True
+
name = entry.name.removesuffix(".tex")
max_len = max(max_len, len(name))
@@ -162,58 +327,100 @@ class SphinxBuilder:
max_len = max(max_len, len(name))
builds[name] = "FAILED (no .tex)"
build_failed = True
+
msg = "Summary"
msg += "\n" + "=" * len(msg)
print()
print(msg)
+
for pdf_name, pdf_file in builds.items():
print(f"{pdf_name:<{max_len}}: {pdf_file}")
+
print()
+
if build_failed:
sys.exit("PDF build failed: not all PDF files were created.")
else:
print("All PDF files were built.")
def handle_info(self, output_dirs):
+ """
+ Extra steps for Info output.
+
+ For texinfo generation, an additional make is needed from the
+ texinfo directory.
+ """
+
for output_dir in output_dirs:
try:
subprocess.run(["make", "info"], cwd=output_dir, check=True)
except subprocess.CalledProcessError as e:
sys.exit(f"Error generating info docs: {e}")
- def cleandocs(self, builder):
+ def cleandocs(self, builder): # pylint: disable=W0613
+ """Remove documentation output directory"""
shutil.rmtree(self.builddir, ignore_errors=True)
def build(self, target, sphinxdirs=None, conf="conf.py",
theme=None, css=None, paper=None):
+ """
+ Build documentation using Sphinx. This is the core function of this
+ module. It prepares all arguments required by sphinx-build.
+ """
+
builder = TARGETS[target]["builder"]
out_dir = TARGETS[target].get("out_dir", "")
+
+ #
+ # Cleandocs doesn't require sphinx-build
+ #
if target == "cleandocs":
self.cleandocs(builder)
return
+
if theme:
- os.environ["DOCS_THEME"] = theme
+ os.environ["DOCS_THEME"] = theme
+
+ #
+ # Other targets require sphinx-build, so check if it exists
+ #
sphinxbuild = shutil.which(self.sphinxbuild, path=self.env["PATH"])
if not sphinxbuild:
sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n")
+
if builder == "latex":
if not self.pdflatex_cmd and not self.latexmk_cmd:
sys.exit("Error: pdflatex or latexmk required for PDF generation")
+
docs_dir = os.path.abspath(os.path.join(self.srctree, "Documentation"))
+
+ #
+ # Fill in base arguments for Sphinx build
+ #
kerneldoc = self.kerneldoc
if kerneldoc.startswith(self.srctree):
kerneldoc = os.path.relpath(kerneldoc, self.srctree)
+
args = [ "-b", builder, "-c", docs_dir ]
+
if builder == "latex":
if not paper:
paper = PAPER[1]
+
args.extend(["-D", f"latex_elements.papersize={paper}paper"])
+
if self.config_rust:
args.extend(["-t", "rustdoc"])
+
if conf:
self.env["SPHINX_CONF"] = self.get_path(conf, abs_path=True)
+
if not sphinxdirs:
sphinxdirs = os.environ.get("SPHINXDIRS", ".")
+
+ #
+ # sphinxdirs can be a list or a whitespace-separated string
+ #
sphinxdirs_list = []
for sphinxdir in sphinxdirs:
if isinstance(sphinxdir, list):
@@ -221,17 +428,32 @@ class SphinxBuilder:
else:
for name in sphinxdir.split(" "):
sphinxdirs_list.append(name)
+
+ #
+ # Step 1: Build each directory in separate.
+ #
+ # This is not the best way of handling it, as cross-references between
+ # them will be broken, but this is what we've been doing since
+ # the beginning.
+ #
output_dirs = []
for sphinxdir in sphinxdirs_list:
src_dir = os.path.join(docs_dir, sphinxdir)
doctree_dir = os.path.join(self.builddir, ".doctrees")
output_dir = os.path.join(self.builddir, sphinxdir, out_dir)
+
+ #
+ # Make directory names canonical
+ #
src_dir = os.path.normpath(src_dir)
doctree_dir = os.path.normpath(doctree_dir)
output_dir = os.path.normpath(output_dir)
+
os.makedirs(doctree_dir, exist_ok=True)
os.makedirs(output_dir, exist_ok=True)
+
output_dirs.append(output_dir)
+
build_args = args + [
"-d", doctree_dir,
"-D", f"kerneldoc_bin={kerneldoc}",
@@ -241,31 +463,54 @@ class SphinxBuilder:
src_dir,
output_dir,
]
+
try:
self.run_sphinx(sphinxbuild, build_args, env=self.env)
except (OSError, ValueError, subprocess.SubprocessError) as e:
sys.exit(f"Build failed: {repr(e)}")
+
+ #
+ # Ensure that each html/epub output will have needed static files
+ #
if target in ["htmldocs", "epubdocs"]:
self.handle_html(css, output_dir)
+
+ #
+ # Step 2: Some targets (PDF and info) require an extra step once
+ # sphinx-build finishes
+ #
if target == "pdfdocs":
self.handle_pdf(output_dirs)
elif target == "infodocs":
self.handle_info(output_dirs)
def jobs_type(value):
+ """
+ Handle valid values for -j. Accepts Sphinx "-jauto", plus a number
+ equal or bigger than one.
+ """
if value is None:
return None
+
if value.lower() == 'auto':
return value.lower()
+
try:
if int(value) >= 1:
return value
+
raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}")
except ValueError:
- raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}")
+ raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}") # pylint: disable=W0707
def main():
+ """
+ Main function. The only mandatory argument is the target. If not
+ specified, the other arguments will use default values if not
+ specified at os.environ.
+ """
parser = argparse.ArgumentParser(description="Kernel documentation builder")
+
parser.add_argument("target", choices=list(TARGETS.keys()),
help="Documentation target to build")
parser.add_argument("--sphinxdirs", nargs="+",
@@ -274,18 +519,27 @@ def main():
help="Sphinx configuration file")
parser.add_argument("--builddir", default="output",
help="Sphinx configuration file")
+
parser.add_argument("--theme", help="Sphinx theme to use")
+
parser.add_argument("--css", help="Custom CSS file for HTML/EPUB")
+
parser.add_argument("--paper", choices=PAPER, default=PAPER[0],
help="Paper size for LaTeX/PDF output")
+
parser.add_argument("-v", "--verbose", action='store_true',
help="place build in verbose mode")
+
parser.add_argument('-j', '--jobs', type=jobs_type,
help="Sets number of jobs to use with sphinx-build")
+
args = parser.parse_args()
+
PythonVersion.check_python(MIN_PYTHON_VERSION)
+
builder = SphinxBuilder(builddir=args.builddir,
verbose=args.verbose, n_jobs=args.jobs)
+
builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
theme=args.theme, css=args.css, paper=args.paper)
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (8 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 09/19] tools/docs: sphinx-build-wrapper: add comments and blank lines Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-10 10:51 ` Jani Nikula
2025-09-04 7:33 ` [PATCH v4 11/19] docs: parallel-wrapper.sh: remove script Mauro Carvalho Chehab
` (9 subsequent siblings)
19 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Sometimes, it is desired to run Sphinx from a virtual environment.
Add a command line parameter to automatically build Sphinx from
such environment.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
1 file changed, 27 insertions(+), 3 deletions(-)
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index ea9f8e17b0bc..cf7b30bc40ff 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
#
# Some constants
#
+VENV_DEFAULT = "sphinx_latest"
MIN_PYTHON_VERSION = PythonVersion("3.7").version
PAPER = ["", "a4", "letter"]
@@ -119,8 +120,9 @@ class SphinxBuilder:
return path
- def __init__(self, builddir, verbose=False, n_jobs=None):
+ def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
"""Initialize internal variables"""
+ self.venv = venv
self.verbose = None
#
@@ -195,6 +197,21 @@ class SphinxBuilder:
self.env = os.environ.copy()
+ #
+ # If venv command line argument is specified, run Sphinx from venv
+ #
+ if venv:
+ bin_dir = os.path.join(venv, "bin")
+ if not os.path.isfile(os.path.join(bin_dir, "activate")):
+ sys.exit(f"Venv {venv} not found.")
+
+ # "activate" virtual env
+ self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
+ self.env["VIRTUAL_ENV"] = venv
+ if "PYTHONHOME" in self.env:
+ del self.env["PYTHONHOME"]
+ print(f"Setting venv to {venv}")
+
def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
"""
Executes sphinx-build using current python3 command and setting
@@ -209,7 +226,10 @@ class SphinxBuilder:
cmd = []
- cmd.append(sys.executable)
+ if self.venv:
+ cmd.append("python")
+ else:
+ cmd.append(sys.executable)
cmd.append(sphinx_build)
@@ -533,11 +553,15 @@ def main():
parser.add_argument('-j', '--jobs', type=jobs_type,
help="Sets number of jobs to use with sphinx-build")
+ parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
+ default=None,
+ help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
+
args = parser.parse_args()
PythonVersion.check_python(MIN_PYTHON_VERSION)
- builder = SphinxBuilder(builddir=args.builddir,
+ builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
verbose=args.verbose, n_jobs=args.jobs)
builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* Re: [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv
2025-09-04 7:33 ` [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv Mauro Carvalho Chehab
@ 2025-09-10 10:51 ` Jani Nikula
2025-09-12 8:46 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-10 10:51 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Sometimes, it is desired to run Sphinx from a virtual environment.
> Add a command line parameter to automatically build Sphinx from
> such environment.
Why?
If you want Sphinx from a virtual environment, you enter the
environment, and run the regular build, with sphinx-build from the PATH
that points at the venv.
We don't do this kind of extra magic for any other tools, I honestly
don't understand why we'd do this for Sphinx. This just adds complexity
for no good reason.
BR,
Jani.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
> 1 file changed, 27 insertions(+), 3 deletions(-)
>
> diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
> index ea9f8e17b0bc..cf7b30bc40ff 100755
> --- a/tools/docs/sphinx-build-wrapper
> +++ b/tools/docs/sphinx-build-wrapper
> @@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
> #
> # Some constants
> #
> +VENV_DEFAULT = "sphinx_latest"
> MIN_PYTHON_VERSION = PythonVersion("3.7").version
> PAPER = ["", "a4", "letter"]
>
> @@ -119,8 +120,9 @@ class SphinxBuilder:
>
> return path
>
> - def __init__(self, builddir, verbose=False, n_jobs=None):
> + def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
> """Initialize internal variables"""
> + self.venv = venv
> self.verbose = None
>
> #
> @@ -195,6 +197,21 @@ class SphinxBuilder:
>
> self.env = os.environ.copy()
>
> + #
> + # If venv command line argument is specified, run Sphinx from venv
> + #
> + if venv:
> + bin_dir = os.path.join(venv, "bin")
> + if not os.path.isfile(os.path.join(bin_dir, "activate")):
> + sys.exit(f"Venv {venv} not found.")
> +
> + # "activate" virtual env
> + self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
> + self.env["VIRTUAL_ENV"] = venv
> + if "PYTHONHOME" in self.env:
> + del self.env["PYTHONHOME"]
> + print(f"Setting venv to {venv}")
> +
> def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
> """
> Executes sphinx-build using current python3 command and setting
> @@ -209,7 +226,10 @@ class SphinxBuilder:
>
> cmd = []
>
> - cmd.append(sys.executable)
> + if self.venv:
> + cmd.append("python")
> + else:
> + cmd.append(sys.executable)
>
> cmd.append(sphinx_build)
>
> @@ -533,11 +553,15 @@ def main():
> parser.add_argument('-j', '--jobs', type=jobs_type,
> help="Sets number of jobs to use with sphinx-build")
>
> + parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
> + default=None,
> + help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
> +
> args = parser.parse_args()
>
> PythonVersion.check_python(MIN_PYTHON_VERSION)
>
> - builder = SphinxBuilder(builddir=args.builddir,
> + builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
> verbose=args.verbose, n_jobs=args.jobs)
>
> builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv
2025-09-10 10:51 ` Jani Nikula
@ 2025-09-12 8:46 ` Mauro Carvalho Chehab
2025-09-12 9:22 ` Jani Nikula
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 8:46 UTC (permalink / raw)
To: Jani Nikula; +Cc: Jonathan Corbet, Linux Doc Mailing List, linux-kernel
Em Wed, 10 Sep 2025 13:51:40 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Sometimes, it is desired to run Sphinx from a virtual environment.
> > Add a command line parameter to automatically build Sphinx from
> > such environment.
>
> Why?
In my case, to be able to test build with different Sphinx versions.
On some distros, only venv works.
> If you want Sphinx from a virtual environment, you enter the
> environment, and run the regular build, with sphinx-build from the PATH
> that points at the venv.
when you do that, ./scripts/spdxcheck.py breaks, affecting checkpatch.
>
> We don't do this kind of extra magic for any other tools, I honestly
> don't understand why we'd do this for Sphinx. This just adds complexity
> for no good reason.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
> > 1 file changed, 27 insertions(+), 3 deletions(-)
> >
> > diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
> > index ea9f8e17b0bc..cf7b30bc40ff 100755
> > --- a/tools/docs/sphinx-build-wrapper
> > +++ b/tools/docs/sphinx-build-wrapper
> > @@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
> > #
> > # Some constants
> > #
> > +VENV_DEFAULT = "sphinx_latest"
> > MIN_PYTHON_VERSION = PythonVersion("3.7").version
> > PAPER = ["", "a4", "letter"]
> >
> > @@ -119,8 +120,9 @@ class SphinxBuilder:
> >
> > return path
> >
> > - def __init__(self, builddir, verbose=False, n_jobs=None):
> > + def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
> > """Initialize internal variables"""
> > + self.venv = venv
> > self.verbose = None
> >
> > #
> > @@ -195,6 +197,21 @@ class SphinxBuilder:
> >
> > self.env = os.environ.copy()
> >
> > + #
> > + # If venv command line argument is specified, run Sphinx from venv
> > + #
> > + if venv:
> > + bin_dir = os.path.join(venv, "bin")
> > + if not os.path.isfile(os.path.join(bin_dir, "activate")):
> > + sys.exit(f"Venv {venv} not found.")
> > +
> > + # "activate" virtual env
> > + self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
> > + self.env["VIRTUAL_ENV"] = venv
> > + if "PYTHONHOME" in self.env:
> > + del self.env["PYTHONHOME"]
> > + print(f"Setting venv to {venv}")
> > +
> > def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
> > """
> > Executes sphinx-build using current python3 command and setting
> > @@ -209,7 +226,10 @@ class SphinxBuilder:
> >
> > cmd = []
> >
> > - cmd.append(sys.executable)
> > + if self.venv:
> > + cmd.append("python")
> > + else:
> > + cmd.append(sys.executable)
> >
> > cmd.append(sphinx_build)
> >
> > @@ -533,11 +553,15 @@ def main():
> > parser.add_argument('-j', '--jobs', type=jobs_type,
> > help="Sets number of jobs to use with sphinx-build")
> >
> > + parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
> > + default=None,
> > + help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
> > +
> > args = parser.parse_args()
> >
> > PythonVersion.check_python(MIN_PYTHON_VERSION)
> >
> > - builder = SphinxBuilder(builddir=args.builddir,
> > + builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
> > verbose=args.verbose, n_jobs=args.jobs)
> >
> > builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
>
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv
2025-09-12 8:46 ` Mauro Carvalho Chehab
@ 2025-09-12 9:22 ` Jani Nikula
2025-09-12 12:34 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-12 9:22 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Jonathan Corbet, Linux Doc Mailing List, linux-kernel
On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Wed, 10 Sep 2025 13:51:40 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
>> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> > Sometimes, it is desired to run Sphinx from a virtual environment.
>> > Add a command line parameter to automatically build Sphinx from
>> > such environment.
>>
>> Why?
>
> In my case, to be able to test build with different Sphinx versions.
> On some distros, only venv works.
I mean why add the complexity of running inside a venv in the wrapper.
>> If you want Sphinx from a virtual environment, you enter the
>> environment, and run the regular build, with sphinx-build from the PATH
>> that points at the venv.
>
> when you do that, ./scripts/spdxcheck.py breaks, affecting checkpatch.
Then you could turn the whole argument around, and say spdxcheck.py
should jump through venv and dependency hoops instead of the docs build.
The point is, it should be the user's responsibility to deal with the
environment and the dependencies.
If they're setting up a virtual environment, and it affects checkpatch,
then they should also include the spdxcheck.py dependencies in the
virtual environment.
This feels like reinventing pipx in a Sphinx wrapper.
We should *reduce* the complexity, not increase it.
>> We don't do this kind of extra magic for any other tools, I honestly
>> don't understand why we'd do this for Sphinx. This just adds complexity
>> for no good reason.
>
>> >
>> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
>> > ---
>> > tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
>> > 1 file changed, 27 insertions(+), 3 deletions(-)
>> >
>> > diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
>> > index ea9f8e17b0bc..cf7b30bc40ff 100755
>> > --- a/tools/docs/sphinx-build-wrapper
>> > +++ b/tools/docs/sphinx-build-wrapper
>> > @@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
>> > #
>> > # Some constants
>> > #
>> > +VENV_DEFAULT = "sphinx_latest"
>> > MIN_PYTHON_VERSION = PythonVersion("3.7").version
>> > PAPER = ["", "a4", "letter"]
>> >
>> > @@ -119,8 +120,9 @@ class SphinxBuilder:
>> >
>> > return path
>> >
>> > - def __init__(self, builddir, verbose=False, n_jobs=None):
>> > + def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
>> > """Initialize internal variables"""
>> > + self.venv = venv
>> > self.verbose = None
>> >
>> > #
>> > @@ -195,6 +197,21 @@ class SphinxBuilder:
>> >
>> > self.env = os.environ.copy()
>> >
>> > + #
>> > + # If venv command line argument is specified, run Sphinx from venv
>> > + #
>> > + if venv:
>> > + bin_dir = os.path.join(venv, "bin")
>> > + if not os.path.isfile(os.path.join(bin_dir, "activate")):
>> > + sys.exit(f"Venv {venv} not found.")
>> > +
>> > + # "activate" virtual env
>> > + self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
>> > + self.env["VIRTUAL_ENV"] = venv
>> > + if "PYTHONHOME" in self.env:
>> > + del self.env["PYTHONHOME"]
>> > + print(f"Setting venv to {venv}")
>> > +
>> > def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
>> > """
>> > Executes sphinx-build using current python3 command and setting
>> > @@ -209,7 +226,10 @@ class SphinxBuilder:
>> >
>> > cmd = []
>> >
>> > - cmd.append(sys.executable)
>> > + if self.venv:
>> > + cmd.append("python")
>> > + else:
>> > + cmd.append(sys.executable)
>> >
>> > cmd.append(sphinx_build)
>> >
>> > @@ -533,11 +553,15 @@ def main():
>> > parser.add_argument('-j', '--jobs', type=jobs_type,
>> > help="Sets number of jobs to use with sphinx-build")
>> >
>> > + parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
>> > + default=None,
>> > + help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
>> > +
>> > args = parser.parse_args()
>> >
>> > PythonVersion.check_python(MIN_PYTHON_VERSION)
>> >
>> > - builder = SphinxBuilder(builddir=args.builddir,
>> > + builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
>> > verbose=args.verbose, n_jobs=args.jobs)
>> >
>> > builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
>>
>
>
>
> Thanks,
> Mauro
>
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv
2025-09-12 9:22 ` Jani Nikula
@ 2025-09-12 12:34 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 12:34 UTC (permalink / raw)
To: Jani Nikula
Cc: Mauro Carvalho Chehab, Jonathan Corbet, Linux Doc Mailing List,
linux-kernel
On Fri, Sep 12, 2025 at 12:22:42PM +0300, Jani Nikula wrote:
> On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Em Wed, 10 Sep 2025 13:51:40 +0300
> > Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> >
> >> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >> > Sometimes, it is desired to run Sphinx from a virtual environment.
> >> > Add a command line parameter to automatically build Sphinx from
> >> > such environment.
> >>
> >> Why?
> >
> > In my case, to be able to test build with different Sphinx versions.
> > On some distros, only venv works.
>
> I mean why add the complexity of running inside a venv in the wrapper.
I don't think it venv support is complex.
>
> >> If you want Sphinx from a virtual environment, you enter the
> >> environment, and run the regular build, with sphinx-build from the PATH
> >> that points at the venv.
> >
> > when you do that, ./scripts/spdxcheck.py breaks, affecting checkpatch.
>
> Then you could turn the whole argument around, and say spdxcheck.py
> should jump through venv and dependency hoops instead of the docs build.
Neither spdxcheck.py nor checkpatch recommends venv; make docs targets do.
> The point is, it should be the user's responsibility to deal with the
> environment and the dependencies.
>
> If they're setting up a virtual environment, and it affects checkpatch,
> then they should also include the spdxcheck.py dependencies in the
> virtual environment.
They are because we're recommending it.
> This feels like reinventing pipx in a Sphinx wrapper.
>
> We should *reduce* the complexity, not increase it.
Complexity is not the issue: There are several things a the Kernel
tree that are complex. Here, the entire wrapper script (not counting
blank lines/comments) is not more than ~300 lines of code, splitted
on multiple functions. This is not complex.
The big issue is what we have now where docs makefile is full of
hacks:
- scripts to workaround issues;
- "|| exit" to fix broken latexmk/xelatex error outputs;
- "+" to use GNU make parallelism;
- complex call macros;
- ...
None of those documented.
Liking or not, this series as a whole makes a lot clearer what
is done to build preparation, Sphinx build and post-build steps
that are required to produce Kernel docs. Also, almost half of
it is documentation. IMHO, a lot better from what we have so
far.
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* [PATCH v4 11/19] docs: parallel-wrapper.sh: remove script
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (9 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 10/19] tools/docs: sphinx-build-wrapper: add support to run inside venv Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter Mauro Carvalho Chehab
` (8 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
The only usage of this script was docs Makefile. Now that
it is using the new sphinx-build-wrapper, which has inside
the code from parallel-wrapper.sh, we can drop this script.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/sphinx/parallel-wrapper.sh | 33 ------------------------
1 file changed, 33 deletions(-)
delete mode 100644 Documentation/sphinx/parallel-wrapper.sh
diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphinx/parallel-wrapper.sh
deleted file mode 100644
index e54c44ce117d..000000000000
--- a/Documentation/sphinx/parallel-wrapper.sh
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0+
-#
-# Figure out if we should follow a specific parallelism from the make
-# environment (as exported by scripts/jobserver-exec), or fall back to
-# the "auto" parallelism when "-jN" is not specified at the top-level
-# "make" invocation.
-
-sphinx="$1"
-shift || true
-
-parallel="$PARALLELISM"
-if [ -z "$parallel" ] ; then
- # If no parallelism is specified at the top-level make, then
- # fall back to the expected "-jauto" mode that the "htmldocs"
- # target has had.
- auto=$(perl -e 'open IN,"'"$sphinx"' --version 2>&1 |";
- while (<IN>) {
- if (m/([\d\.]+)/) {
- print "auto" if ($1 >= "1.7")
- }
- }
- close IN')
- if [ -n "$auto" ] ; then
- parallel="$auto"
- fi
-fi
-# Only if some parallelism has been determined do we add the -jN option.
-if [ -n "$parallel" ] ; then
- parallel="-j$parallel"
-fi
-
-exec "$sphinx" $parallel "$@"
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (10 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 11/19] docs: parallel-wrapper.sh: remove script Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-10 10:54 ` Jani Nikula
2025-09-04 7:33 ` [PATCH v4 13/19] tools/docs: sphinx-build-wrapper: add an argument for LaTeX interactive mode Mauro Carvalho Chehab
` (7 subsequent siblings)
19 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
While the build system supports this for a long time, this was
never documented. Add a documentation for it.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 4736f02b6c9e..0e1d8657a5cc 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -126,4 +126,6 @@ dochelp:
@echo
@echo ' make DOCS_CSS={a .css file} adds a DOCS_CSS override file for html/epub output.'
@echo
+ @echo ' make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
+ @echo
@echo ' Default location for the generated documents is Documentation/output'
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* Re: [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter
2025-09-04 7:33 ` [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter Mauro Carvalho Chehab
@ 2025-09-10 10:54 ` Jani Nikula
2025-09-12 8:56 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-10 10:54 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> While the build system supports this for a long time, this was
> never documented. Add a documentation for it.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> Documentation/Makefile | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 4736f02b6c9e..0e1d8657a5cc 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -126,4 +126,6 @@ dochelp:
> @echo
> @echo ' make DOCS_CSS={a .css file} adds a DOCS_CSS override file for html/epub output.'
> @echo
> + @echo ' make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
> + @echo
> @echo ' Default location for the generated documents is Documentation/output'
The Sphinx make mode 'sphinx-build -M help' provides all of this and
more...
BR,
Jani.
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter
2025-09-10 10:54 ` Jani Nikula
@ 2025-09-12 8:56 ` Mauro Carvalho Chehab
2025-09-12 9:23 ` Jani Nikula
0 siblings, 1 reply; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 8:56 UTC (permalink / raw)
To: Jani Nikula; +Cc: Jonathan Corbet, Linux Doc Mailing List, linux-kernel
Em Wed, 10 Sep 2025 13:54:07 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > While the build system supports this for a long time, this was
> > never documented. Add a documentation for it.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > Documentation/Makefile | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/Documentation/Makefile b/Documentation/Makefile
> > index 4736f02b6c9e..0e1d8657a5cc 100644
> > --- a/Documentation/Makefile
> > +++ b/Documentation/Makefile
> > @@ -126,4 +126,6 @@ dochelp:
> > @echo
> > @echo ' make DOCS_CSS={a .css file} adds a DOCS_CSS override file for html/epub output.'
> > @echo
> > + @echo ' make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
> > + @echo
> > @echo ' Default location for the generated documents is Documentation/output'
>
> The Sphinx make mode 'sphinx-build -M help' provides all of this and
> more...
So? Relying on a help message from sphinx-build would just make
our makefile fragile.
btw, it didn't work here:
$ ./sphinx_latest/bin/sphinx-build --version
sphinx-build 8.2.3
$ ./sphinx_latest/bin/sphinx-build -M help
Error: at least 3 arguments (builder, source dir, build dir) are required.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter
2025-09-12 8:56 ` Mauro Carvalho Chehab
@ 2025-09-12 9:23 ` Jani Nikula
2025-09-12 10:34 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 67+ messages in thread
From: Jani Nikula @ 2025-09-12 9:23 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Jonathan Corbet, Linux Doc Mailing List, linux-kernel
On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Wed, 10 Sep 2025 13:54:07 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
>> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> > While the build system supports this for a long time, this was
>> > never documented. Add a documentation for it.
>> >
>> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
>> > ---
>> > Documentation/Makefile | 2 ++
>> > 1 file changed, 2 insertions(+)
>> >
>> > diff --git a/Documentation/Makefile b/Documentation/Makefile
>> > index 4736f02b6c9e..0e1d8657a5cc 100644
>> > --- a/Documentation/Makefile
>> > +++ b/Documentation/Makefile
>> > @@ -126,4 +126,6 @@ dochelp:
>> > @echo
>> > @echo ' make DOCS_CSS={a .css file} adds a DOCS_CSS override file for html/epub output.'
>> > @echo
>> > + @echo ' make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
>> > + @echo
>> > @echo ' Default location for the generated documents is Documentation/output'
>>
>> The Sphinx make mode 'sphinx-build -M help' provides all of this and
>> more...
>
> So? Relying on a help message from sphinx-build would just make
> our makefile fragile.
>
> btw, it didn't work here:
>
> $ ./sphinx_latest/bin/sphinx-build --version
> sphinx-build 8.2.3
>
> $ ./sphinx_latest/bin/sphinx-build -M help
> Error: at least 3 arguments (builder, source dir, build dir) are required.
'sphinx-build -M help . .'
>
> Thanks,
> Mauro
--
Jani Nikula, Intel
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter
2025-09-12 9:23 ` Jani Nikula
@ 2025-09-12 10:34 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-12 10:34 UTC (permalink / raw)
To: Jani Nikula; +Cc: Jonathan Corbet, Linux Doc Mailing List, linux-kernel
Em Fri, 12 Sep 2025 12:23:38 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Em Wed, 10 Sep 2025 13:54:07 +0300
> > Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> >
> >> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >> > While the build system supports this for a long time, this was
> >> > never documented. Add a documentation for it.
> >> >
> >> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> >> > ---
> >> > Documentation/Makefile | 2 ++
> >> > 1 file changed, 2 insertions(+)
> >> >
> >> > diff --git a/Documentation/Makefile b/Documentation/Makefile
> >> > index 4736f02b6c9e..0e1d8657a5cc 100644
> >> > --- a/Documentation/Makefile
> >> > +++ b/Documentation/Makefile
> >> > @@ -126,4 +126,6 @@ dochelp:
> >> > @echo
> >> > @echo ' make DOCS_CSS={a .css file} adds a DOCS_CSS override file for html/epub output.'
> >> > @echo
> >> > + @echo ' make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
> >> > + @echo
> >> > @echo ' Default location for the generated documents is Documentation/output'
> >>
> >> The Sphinx make mode 'sphinx-build -M help' provides all of this and
> >> more...
> >
> > So? Relying on a help message from sphinx-build would just make
> > our makefile fragile.
> >
> > btw, it didn't work here:
> >
> > $ ./sphinx_latest/bin/sphinx-build --version
> > sphinx-build 8.2.3
> >
> > $ ./sphinx_latest/bin/sphinx-build -M help
> > Error: at least 3 arguments (builder, source dir, build dir) are required.
>
> 'sphinx-build -M help . .'
I'm still missing the point:
$ ./sphinx_latest/bin/sphinx-build -M help . .
Sphinx v8.2.3
Please use `make target' where target is one of
html to make standalone HTML files
dirhtml to make HTML files named index.html in directories
singlehtml to make a single large HTML file
pickle to make pickle files
json to make JSON files
htmlhelp to make HTML files and an HTML help project
qthelp to make HTML files and a qthelp project
devhelp to make HTML files and a Devhelp project
epub to make an epub
latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter
latexpdf to make LaTeX and PDF files (default pdflatex)
latexpdfja to make LaTeX files and run them through platex/dvipdfmx
text to make text files
man to make manual pages
texinfo to make Texinfo files
info to make Texinfo files and run them through makeinfo
gettext to make PO message catalogs
changes to make an overview of all changed/added/deprecated items
xml to make Docutils-native XML files
pseudoxml to make pseudoxml-XML files for display purposes
linkcheck to check all external links for integrity
doctest to run all doctests embedded in the documentation (if enabled)
coverage to run coverage check of the documentation (if enabled)
clean to remove everything in the build directory
Why is it related to this change?
+ @echo ' make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
Regards,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread
* [PATCH v4 13/19] tools/docs: sphinx-build-wrapper: add an argument for LaTeX interactive mode
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (11 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 12/19] docs: Makefile: document latex/PDF PAPER= parameter Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 14/19] tools/docs,scripts: sphinx-*: prevent sphinx-build crashes Mauro Carvalho Chehab
` (6 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
By default, we use LaTeX batch mode to build docs. This way, when
an error happens, the build fails. This is good for normal builds,
but when debugging problems with pdf generation, the best is to
use interactive mode.
We already support it via LATEXOPTS, but having a command line
argument makes it easier:
Interactive mode:
./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v -i
...
Running 'xelatex --no-pdf -no-pdf -recorder ".../Documentation/output/peci/latex/peci.tex"'
...
Default batch mode:
./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v
...
Running 'xelatex --no-pdf -no-pdf -interaction=batchmode -no-shell-escape -recorder ".../Documentation/output/peci/latex/peci.tex"'
...
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/sphinx-build-wrapper | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index cf7b30bc40ff..8ce57a25696f 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -120,7 +120,8 @@ class SphinxBuilder:
return path
- def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
+ def __init__(self, builddir, venv=None, verbose=False, n_jobs=None,
+ interactive=None):
"""Initialize internal variables"""
self.venv = venv
self.verbose = None
@@ -131,7 +132,11 @@ class SphinxBuilder:
self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
- self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+
+ if not interactive:
+ self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+ else:
+ self.latexopts = os.environ.get("LATEXOPTS", "")
if not verbose:
verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
@@ -553,6 +558,9 @@ def main():
parser.add_argument('-j', '--jobs', type=jobs_type,
help="Sets number of jobs to use with sphinx-build")
+ parser.add_argument('-i', '--interactive', action='store_true',
+ help="Change latex default to run in interactive mode")
+
parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
default=None,
help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
@@ -562,7 +570,8 @@ def main():
PythonVersion.check_python(MIN_PYTHON_VERSION)
builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
- verbose=args.verbose, n_jobs=args.jobs)
+ verbose=args.verbose, n_jobs=args.jobs,
+ interactive=args.interactive)
builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
theme=args.theme, css=args.css, paper=args.paper)
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 14/19] tools/docs,scripts: sphinx-*: prevent sphinx-build crashes
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (12 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 13/19] tools/docs: sphinx-build-wrapper: add an argument for LaTeX interactive mode Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 15/19] tools/docs: sphinx-build-wrapper: allow building PDF files in parallel Mauro Carvalho Chehab
` (5 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
On a properly set system, LANG and LC_ALL is always defined.
However, some distros like Debian, Gentoo and their variants
start with those undefioned.
When Sphinx tries to set a locale with:
locale.setlocale(locale.LC_ALL, '')
It raises an exception, making Sphinx fail. This is more likely
to happen with test containers.
Add a logic to detect and workaround such issue by setting
locale to C.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/sphinx-build-wrapper | 11 +++++++++++
tools/docs/sphinx-pre-install | 14 +++++++++++++-
2 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index 8ce57a25696f..c318af5f2af1 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -45,6 +45,7 @@ the newer version.
"""
import argparse
+import locale
import os
import shlex
import shutil
@@ -443,6 +444,16 @@ class SphinxBuilder:
if not sphinxdirs:
sphinxdirs = os.environ.get("SPHINXDIRS", ".")
+ #
+ # The sphinx-build tool has a bug: internally, it tries to set
+ # locale with locale.setlocale(locale.LC_ALL, ''). This causes a
+ # crash if language is not set. Detect and fix it.
+ #
+ try:
+ locale.setlocale(locale.LC_ALL, '')
+ except locale.Error:
+ self.env["LC_ALL"] = "C"
+
#
# sphinxdirs can be a list or a whitespace-separated string
#
diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install
index d6d673b7945c..663d4e2a3f57 100755
--- a/tools/docs/sphinx-pre-install
+++ b/tools/docs/sphinx-pre-install
@@ -26,6 +26,7 @@ system pacage install is recommended.
"""
import argparse
+import locale
import os
import re
import subprocess
@@ -422,8 +423,19 @@ class MissingCheckers(AncillaryMethods):
"""
Gets sphinx-build version.
"""
+ env = os.environ.copy()
+
+ # The sphinx-build tool has a bug: internally, it tries to set
+ # locale with locale.setlocale(locale.LC_ALL, ''). This causes a
+ # crash if language is not set. Detect and fix it.
try:
- result = self.run([cmd, "--version"],
+ locale.setlocale(locale.LC_ALL, '')
+ except Exception:
+ env["LC_ALL"] = "C"
+ env["LANG"] = "C"
+
+ try:
+ result = self.run([cmd, "--version"], env=env,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True, check=True)
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 15/19] tools/docs: sphinx-build-wrapper: allow building PDF files in parallel
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (13 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 14/19] tools/docs,scripts: sphinx-*: prevent sphinx-build crashes Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 16/19] docs: add support to build manpages from kerneldoc output Mauro Carvalho Chehab
` (4 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Use POSIX jobserver when available or -j<number> to run PDF
builds in parallel, restoring pdf build performance. Yet,
running it when debugging troubles is a bad idea, so, when
calling directly via command line, except if "-j" is splicitly
requested, it will serialize the build.
With such change, a PDF doc builds now takes around 5 minutes
on a Ryzen 9 machine with 32 cpu threads:
# Explicitly paralelize both Sphinx and LaTeX pdf builds
$ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs -j 33
real 5m17.901s
user 15m1.499s
sys 2m31.482s
# Use POSIX jobserver to paralelize both sphinx-build and LaTeX
$ make cleandocs; time make pdfdocs
real 5m22.369s
user 15m9.076s
sys 2m31.419s
# Serializes PDF build, while keeping Sphinx parallelized.
# it is equivalent of passing -jauto via command line
$ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs
real 11m20.901s
user 13m2.910s
sys 1m44.553s
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/sphinx-build-wrapper | 158 +++++++++++++++++++++++---------
1 file changed, 117 insertions(+), 41 deletions(-)
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index c318af5f2af1..c5fcc448a275 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -52,6 +52,8 @@ import shutil
import subprocess
import sys
+from concurrent import futures
+
from lib.python_version import PythonVersion
LIB_DIR = "../../scripts/lib"
@@ -282,6 +284,82 @@ class SphinxBuilder:
except (OSError, IOError) as e:
print(f"Warning: Failed to copy CSS: {e}", file=sys.stderr)
+ def build_pdf_file(self, latex_cmd, from_dir, path):
+ """Builds a single pdf file using latex_cmd"""
+ try:
+ subprocess.run(latex_cmd + [path],
+ cwd=from_dir, check=True)
+
+ return True
+ except subprocess.CalledProcessError:
+ return False
+
+ def pdf_parallel_build(self, tex_suffix, latex_cmd, tex_files, n_jobs):
+ """Build PDF files in parallel if possible"""
+ builds = {}
+ build_failed = False
+ max_len = 0
+ has_tex = False
+
+ #
+ # LaTeX PDF error code is almost useless for us:
+ # any warning makes it non-zero. For kernel doc builds it always return
+ # non-zero even when build succeeds. So, let's do the best next thing:
+ # Ignore build errors. At the end, check if all PDF files were built,
+ # printing a summary with the built ones and returning 0 if all of
+ # them were actually built.
+ #
+ with futures.ThreadPoolExecutor(max_workers=n_jobs) as executor:
+ jobs = {}
+
+ for from_dir, pdf_dir, entry in tex_files:
+ name = entry.name
+
+ if not name.endswith(tex_suffix):
+ continue
+
+ name = name[:-len(tex_suffix)]
+
+ max_len = max(max_len, len(name))
+
+ has_tex = True
+
+ future = executor.submit(self.build_pdf_file, latex_cmd,
+ from_dir, entry.path)
+ jobs[future] = (from_dir, pdf_dir, name)
+
+ for future in futures.as_completed(jobs):
+ from_dir, pdf_dir, name = jobs[future]
+
+ pdf_name = name + ".pdf"
+ pdf_from = os.path.join(from_dir, pdf_name)
+
+ try:
+ success = future.result()
+
+ if success and os.path.exists(pdf_from):
+ pdf_to = os.path.join(pdf_dir, pdf_name)
+
+ os.rename(pdf_from, pdf_to)
+ builds[name] = os.path.relpath(pdf_to, self.builddir)
+ else:
+ builds[name] = "FAILED"
+ build_failed = True
+ except futures.Error as e:
+ builds[name] = f"FAILED ({repr(e)})"
+ build_failed = True
+
+ #
+ # Handle case where no .tex files were found
+ #
+ if not has_tex:
+ name = "Sphinx LaTeX builder"
+ max_len = max(max_len, len(name))
+ builds[name] = "FAILED (no .tex file was generated)"
+ build_failed = True
+
+ return builds, build_failed, max_len
+
def handle_pdf(self, output_dirs):
"""
Extra steps for PDF output.
@@ -292,7 +370,9 @@ class SphinxBuilder:
"""
builds = {}
max_len = 0
+ tex_suffix = ".tex"
+ tex_files = []
for from_dir in output_dirs:
pdf_dir = os.path.join(from_dir, "../pdf")
os.makedirs(pdf_dir, exist_ok=True)
@@ -304,55 +384,51 @@ class SphinxBuilder:
latex_cmd.extend(shlex.split(self.latexopts))
- tex_suffix = ".tex"
-
- #
- # Process each .tex file
- #
-
- has_tex = False
- build_failed = False
+ # Get a list of tex files to process
with os.scandir(from_dir) as it:
for entry in it:
- if not entry.name.endswith(tex_suffix):
- continue
+ if entry.name.endswith(tex_suffix):
+ tex_files.append((from_dir, pdf_dir, entry))
- name = entry.name[:-len(tex_suffix)]
- has_tex = True
+ #
+ # When using make, this won't be used, as the number of jobs comes
+ # from POSIX jobserver. So, this covers the case where build comes
+ # from command line. On such case, serialize by default, except if
+ # the user explicitly sets the number of jobs.
+ #
+ n_jobs = 1
- #
- # LaTeX PDF error code is almost useless for us:
- # any warning makes it non-zero. For kernel doc builds it
- # always return non-zero even when build succeeds.
- # So, let's do the best next thing: check if all PDF
- # files were built. If they're, print a summary and
- # return 0 at the end of this function
- #
- try:
- subprocess.run(latex_cmd + [entry.path],
- cwd=from_dir, check=True)
- except subprocess.CalledProcessError:
- pass
+ # n_jobs is either an integer or "auto". Only use it if it is a number
+ if self.n_jobs:
+ try:
+ n_jobs = int(self.n_jobs)
+ except ValueError:
+ pass
- pdf_name = name + ".pdf"
- pdf_from = os.path.join(from_dir, pdf_name)
- pdf_to = os.path.join(pdf_dir, pdf_name)
+ #
+ # When using make, jobserver.claim is the number of jobs that were
+ # used with "-j" and that aren't used by other make targets
+ #
+ with JobserverExec() as jobserver:
+ n_jobs = 1
- if os.path.exists(pdf_from):
- os.rename(pdf_from, pdf_to)
- builds[name] = os.path.relpath(pdf_to, self.builddir)
- else:
- builds[name] = "FAILED"
- build_failed = True
+ #
+ # Handle the case when a parameter is passed via command line,
+ # using it as default, if jobserver doesn't claim anything
+ #
+ if self.n_jobs:
+ try:
+ n_jobs = int(self.n_jobs)
+ except ValueError:
+ pass
- name = entry.name.removesuffix(".tex")
- max_len = max(max_len, len(name))
+ if jobserver.claim:
+ n_jobs = jobserver.claim
- if not has_tex:
- name = os.path.basename(from_dir)
- max_len = max(max_len, len(name))
- builds[name] = "FAILED (no .tex)"
- build_failed = True
+ builds, build_failed, max_len = self.pdf_parallel_build(tex_suffix,
+ latex_cmd,
+ tex_files,
+ n_jobs)
msg = "Summary"
msg += "\n" + "=" * len(msg)
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 16/19] docs: add support to build manpages from kerneldoc output
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (14 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 15/19] tools/docs: sphinx-build-wrapper: allow building PDF files in parallel Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 17/19] tools: kernel-doc: add a see also section at man pages Mauro Carvalho Chehab
` (3 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Thomas Weißschuh, Alice Ryhl,
Masahiro Yamada, Mauro Carvalho Chehab, Miguel Ojeda,
Nathan Chancellor, Nicolas Schier, Randy Dunlap, Tamir Duberstein,
linux-kbuild, linux-kernel
Generating man files currently requires running a separate
script. The target also doesn't appear at the docs Makefile.
Add support for mandocs at the Makefile, adding the build
logic inside sphinx-build-wrapper, updating documentation
and dropping the ancillary script.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 3 +-
Documentation/doc-guide/kernel-doc.rst | 29 ++++-----
Makefile | 2 +-
scripts/split-man.pl | 28 ---------
tools/docs/sphinx-build-wrapper | 81 ++++++++++++++++++++++++--
5 files changed, 95 insertions(+), 48 deletions(-)
delete mode 100755 scripts/split-man.pl
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 0e1d8657a5cc..f9b6e9386a58 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -53,7 +53,7 @@ ifeq ($(HAVE_SPHINX),0)
else # HAVE_SPHINX
# Common documentation targets
-infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
+mandocs infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
$(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check
+$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
--sphinxdirs="$(SPHINXDIRS)" --conf="$(SPHINX_CONF)" \
@@ -106,6 +106,7 @@ dochelp:
@echo ' htmldocs - HTML'
@echo ' texinfodocs - Texinfo'
@echo ' infodocs - Info'
+ @echo ' mandocs - Man pages'
@echo ' latexdocs - LaTeX'
@echo ' pdfdocs - PDF'
@echo ' epubdocs - EPUB'
diff --git a/Documentation/doc-guide/kernel-doc.rst b/Documentation/doc-guide/kernel-doc.rst
index af9697e60165..4370cc8fbcf5 100644
--- a/Documentation/doc-guide/kernel-doc.rst
+++ b/Documentation/doc-guide/kernel-doc.rst
@@ -579,20 +579,23 @@ source.
How to use kernel-doc to generate man pages
-------------------------------------------
-If you just want to use kernel-doc to generate man pages you can do this
-from the kernel git tree::
+To generate man pages for all files that contain kernel-doc markups, run::
- $ scripts/kernel-doc -man \
- $(git grep -l '/\*\*' -- :^Documentation :^tools) \
- | scripts/split-man.pl /tmp/man
+ $ make mandocs
-Some older versions of git do not support some of the variants of syntax for
-path exclusion. One of the following commands may work for those versions::
+Or calling ``script-build-wrapper`` directly::
- $ scripts/kernel-doc -man \
- $(git grep -l '/\*\*' -- . ':!Documentation' ':!tools') \
- | scripts/split-man.pl /tmp/man
+ $ ./tools/docs/sphinx-build-wrapper mandocs
- $ scripts/kernel-doc -man \
- $(git grep -l '/\*\*' -- . ":(exclude)Documentation" ":(exclude)tools") \
- | scripts/split-man.pl /tmp/man
+The output will be at ``/man`` directory inside the output directory
+(by default: ``Documentation/output``).
+
+Optionally, it is possible to generate a partial set of man pages by
+using SPHINXDIRS:
+
+ $ make SPHINXDIRS=driver-api/media mandocs
+
+.. note::
+
+ When SPHINXDIRS={subdir} is used, it will only generate man pages for
+ the files explicitly inside a ``Documentation/{subdir}/.../*.rst`` file.
diff --git a/Makefile b/Makefile
index 6bfe776bf3c5..9bd44afeda26 100644
--- a/Makefile
+++ b/Makefile
@@ -1800,7 +1800,7 @@ $(help-board-dirs): help-%:
# Documentation targets
# ---------------------------------------------------------------------------
DOC_TARGETS := xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \
- linkcheckdocs dochelp refcheckdocs texinfodocs infodocs
+ linkcheckdocs dochelp refcheckdocs texinfodocs infodocs mandocs
PHONY += $(DOC_TARGETS)
$(DOC_TARGETS):
$(Q)$(MAKE) $(build)=Documentation $@
diff --git a/scripts/split-man.pl b/scripts/split-man.pl
deleted file mode 100755
index 96bd99dc977a..000000000000
--- a/scripts/split-man.pl
+++ /dev/null
@@ -1,28 +0,0 @@
-#!/usr/bin/env perl
-# SPDX-License-Identifier: GPL-2.0
-#
-# Author: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
-#
-# Produce manpages from kernel-doc.
-# See Documentation/doc-guide/kernel-doc.rst for instructions
-
-if ($#ARGV < 0) {
- die "where do I put the results?\n";
-}
-
-mkdir $ARGV[0],0777;
-$state = 0;
-while (<STDIN>) {
- if (/^\.TH \"[^\"]*\" 9 \"([^\"]*)\"/) {
- if ($state == 1) { close OUT }
- $state = 1;
- $fn = "$ARGV[0]/$1.9";
- print STDERR "Creating $fn\n";
- open OUT, ">$fn" or die "can't open $fn: $!\n";
- print OUT $_;
- } elsif ($state != 0) {
- print OUT $_;
- }
-}
-
-close OUT;
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index c5fcc448a275..183717eb6d87 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -47,6 +47,7 @@ the newer version.
import argparse
import locale
import os
+import re
import shlex
import shutil
import subprocess
@@ -55,6 +56,7 @@ import sys
from concurrent import futures
from lib.python_version import PythonVersion
+from glob import glob
LIB_DIR = "../../scripts/lib"
SRC_DIR = os.path.dirname(os.path.realpath(__file__))
@@ -77,6 +79,7 @@ TARGETS = {
"epubdocs": { "builder": "epub", "out_dir": "epub" },
"texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" },
"infodocs": { "builder": "texinfo", "out_dir": "texinfo" },
+ "mandocs": { "builder": "man", "out_dir": "man" },
"latexdocs": { "builder": "latex", "out_dir": "latex" },
"pdfdocs": { "builder": "latex", "out_dir": "latex" },
"xmldocs": { "builder": "xml", "out_dir": "xml" },
@@ -459,6 +462,71 @@ class SphinxBuilder:
except subprocess.CalledProcessError as e:
sys.exit(f"Error generating info docs: {e}")
+ def handle_man(self, kerneldoc, docs_dir, src_dir, output_dir):
+ """
+ Create man pages from kernel-doc output
+ """
+
+ re_kernel_doc = re.compile(r"^\.\.\s+kernel-doc::\s*(\S+)")
+ re_man = re.compile(r'^\.TH "[^"]*" (\d+) "([^"]*)"')
+
+ if docs_dir == src_dir:
+ #
+ # Pick the entire set of kernel-doc markups from the entire tree
+ #
+ kdoc_files = set([self.srctree])
+ else:
+ kdoc_files = set()
+
+ for fname in glob(os.path.join(src_dir, "**"), recursive=True):
+ if os.path.isfile(fname) and fname.endswith(".rst"):
+ with open(fname, "r", encoding="utf-8") as in_fp:
+ data = in_fp.read()
+
+ for line in data.split("\n"):
+ match = re_kernel_doc.match(line)
+ if match:
+ if os.path.isfile(match.group(1)):
+ kdoc_files.add(match.group(1))
+
+ if not kdoc_files:
+ sys.exit(f"Directory {src_dir} doesn't contain kernel-doc tags")
+
+ cmd = [ kerneldoc, "-m" ] + sorted(kdoc_files)
+ try:
+ if self.verbose:
+ print(" ".join(cmd))
+
+ result = subprocess.run(cmd, stdout=subprocess.PIPE, text= True)
+
+ if result.returncode:
+ print(f"Warning: kernel-doc returned {result.returncode} warnings")
+
+ except (OSError, ValueError, subprocess.SubprocessError) as e:
+ sys.exit(f"Failed to create man pages for {src_dir}: {repr(e)}")
+
+ fp = None
+ try:
+ for line in result.stdout.split("\n"):
+ match = re_man.match(line)
+ if not match:
+ if fp:
+ fp.write(line + '\n')
+ continue
+
+ if fp:
+ fp.close()
+
+ fname = f"{output_dir}/{match.group(2)}.{match.group(1)}"
+
+ if self.verbose:
+ print(f"Creating {fname}")
+ fp = open(fname, "w", encoding="utf-8")
+ fp.write(line + '\n')
+ finally:
+ if fp:
+ fp.close()
+
def cleandocs(self, builder): # pylint: disable=W0613
"""Remove documentation output directory"""
shutil.rmtree(self.builddir, ignore_errors=True)
@@ -487,7 +555,7 @@ class SphinxBuilder:
# Other targets require sphinx-build, so check if it exists
#
sphinxbuild = shutil.which(self.sphinxbuild, path=self.env["PATH"])
- if not sphinxbuild:
+ if not sphinxbuild and target != "mandocs":
sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n")
if builder == "latex":
@@ -576,10 +644,13 @@ class SphinxBuilder:
output_dir,
]
- try:
- self.run_sphinx(sphinxbuild, build_args, env=self.env)
- except (OSError, ValueError, subprocess.SubprocessError) as e:
- sys.exit(f"Build failed: {repr(e)}")
+ if target == "mandocs":
+ self.handle_man(kerneldoc, docs_dir, src_dir, output_dir)
+ else:
+ try:
+ self.run_sphinx(sphinxbuild, build_args, env=self.env)
+ except (OSError, ValueError, subprocess.SubprocessError) as e:
+ sys.exit(f"Build failed: {repr(e)}")
#
# Ensure that each html/epub output will have needed static files
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 17/19] tools: kernel-doc: add a see also section at man pages
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (15 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 16/19] docs: add support to build manpages from kerneldoc output Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 18/19] scripts: kdoc_parser.py: warn about Python version only once Mauro Carvalho Chehab
` (2 subsequent siblings)
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
While cross-references are complex, as related ones can be on
different files, we can at least correlate the ones that belong
to the same file, adding a SEE ALSO section for them.
The result is not bad. See for instance:
$ tools/docs/sphinx-build-wrapper --sphinxdirs driver-api/media -- mandocs
$ man Documentation/output/driver-api/man/edac_pci_add_device.9
edac_pci_add_device(9) Kernel Hacker's Manual edac_pci_add_device(9)
NAME
edac_pci_add_device - Insert the 'edac_dev' structure into the
edac_pci global list and create sysfs entries associated with
edac_pci structure.
SYNOPSIS
int edac_pci_add_device (struct edac_pci_ctl_info *pci , int
edac_idx );
ARGUMENTS
pci pointer to the edac_device structure to be added to
the list
edac_idx A unique numeric identifier to be assigned to the
RETURN
0 on Success, or an error code on failure
SEE ALSO
edac_pci_alloc_ctl_info(9), edac_pci_free_ctl_info(9),
edac_pci_alloc_index(9), edac_pci_del_device(9), edac_pci_cre‐
ate_generic_ctl(9), edac_pci_release_generic_ctl(9),
edac_pci_create_sysfs(9), edac_pci_remove_sysfs(9)
August 2025 edac_pci_add_device edac_pci_add_device(9)
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/lib/kdoc/kdoc_files.py | 5 +-
scripts/lib/kdoc/kdoc_output.py | 84 +++++++++++++++++++++++++++++++--
2 files changed, 83 insertions(+), 6 deletions(-)
diff --git a/scripts/lib/kdoc/kdoc_files.py b/scripts/lib/kdoc/kdoc_files.py
index 9e09b45b02fa..061c033f32da 100644
--- a/scripts/lib/kdoc/kdoc_files.py
+++ b/scripts/lib/kdoc/kdoc_files.py
@@ -275,7 +275,10 @@ class KernelFiles():
self.config.log.warning("No kernel-doc for file %s", fname)
continue
- for arg in self.results[fname]:
+ symbols = self.results[fname]
+ self.out_style.set_symbols(symbols)
+
+ for arg in symbols:
m = self.out_msg(fname, arg.name, arg)
if m is None:
diff --git a/scripts/lib/kdoc/kdoc_output.py b/scripts/lib/kdoc/kdoc_output.py
index ea8914537ba0..1eca9a918558 100644
--- a/scripts/lib/kdoc/kdoc_output.py
+++ b/scripts/lib/kdoc/kdoc_output.py
@@ -215,6 +215,9 @@ class OutputFormat:
# Virtual methods to be overridden by inherited classes
# At the base class, those do nothing.
+ def set_symbols(self, symbols):
+ """Get a list of all symbols from kernel_doc"""
+
def out_doc(self, fname, name, args):
"""Outputs a DOC block"""
@@ -577,6 +580,7 @@ class ManFormat(OutputFormat):
super().__init__()
self.modulename = modulename
+ self.symbols = []
dt = None
tstamp = os.environ.get("KBUILD_BUILD_TIMESTAMP")
@@ -593,6 +597,68 @@ class ManFormat(OutputFormat):
self.man_date = dt.strftime("%B %Y")
+ def arg_name(self, args, name):
+ """
+ Return the name that will be used for the man page.
+
+ As we may have the same name on different namespaces,
+ prepend the data type for all types except functions and typedefs.
+
+ The doc section is special: it uses the modulename.
+ """
+
+ dtype = args.type
+
+ if dtype == "doc":
+ return self.modulename
+
+ if dtype in ["function", "typedef"]:
+ return name
+
+ return f"{dtype} {name}"
+
+ def set_symbols(self, symbols):
+ """
+ Get a list of all symbols from kernel_doc.
+
+ Man pages will uses it to add a SEE ALSO section with other
+ symbols at the same file.
+ """
+ self.symbols = symbols
+
+ def out_tail(self, fname, name, args):
+ """Adds a tail for all man pages"""
+
+ # SEE ALSO section
+ if len(self.symbols) >= 2:
+ cur_name = self.arg_name(args, name)
+
+ self.data += f'.SH "SEE ALSO"' + "\n.PP\n"
+ related = []
+ for arg in self.symbols:
+ out_name = self.arg_name(arg, arg.name)
+
+ if cur_name == out_name:
+ continue
+
+ related.append(f"\\fB{out_name}\\fR(9)")
+
+ self.data += ",\n".join(related) + "\n"
+
+ # TODO: does it make sense to add other sections? Maybe
+ # REPORTING ISSUES? LICENSE?
+
+ def msg(self, fname, name, args):
+ """
+ Handles a single entry from kernel-doc parser.
+
+ Add a tail at the end of man pages output.
+ """
+ super().msg(fname, name, args)
+ self.out_tail(fname, name, args)
+
+ return self.data
+
def output_highlight(self, block):
"""
Outputs a C symbol that may require being highlighted with
@@ -618,7 +684,9 @@ class ManFormat(OutputFormat):
if not self.check_doc(name, args):
return
- self.data += f'.TH "{self.modulename}" 9 "{self.modulename}" "{self.man_date}" "API Manual" LINUX' + "\n"
+ out_name = self.arg_name(args, name)
+
+ self.data += f'.TH "{self.modulename}" 9 "{out_name}" "{self.man_date}" "API Manual" LINUX' + "\n"
for section, text in args.sections.items():
self.data += f'.SH "{section}"' + "\n"
@@ -627,7 +695,9 @@ class ManFormat(OutputFormat):
def out_function(self, fname, name, args):
"""output function in man"""
- self.data += f'.TH "{name}" 9 "{name}" "{self.man_date}" "Kernel Hacker\'s Manual" LINUX' + "\n"
+ out_name = self.arg_name(args, name)
+
+ self.data += f'.TH "{name}" 9 "{out_name}" "{self.man_date}" "Kernel Hacker\'s Manual" LINUX' + "\n"
self.data += ".SH NAME\n"
self.data += f"{name} \\- {args['purpose']}\n"
@@ -671,7 +741,9 @@ class ManFormat(OutputFormat):
self.output_highlight(text)
def out_enum(self, fname, name, args):
- self.data += f'.TH "{self.modulename}" 9 "enum {name}" "{self.man_date}" "API Manual" LINUX' + "\n"
+ out_name = self.arg_name(args, name)
+
+ self.data += f'.TH "{self.modulename}" 9 "{out_name}" "{self.man_date}" "API Manual" LINUX' + "\n"
self.data += ".SH NAME\n"
self.data += f"enum {name} \\- {args['purpose']}\n"
@@ -703,8 +775,9 @@ class ManFormat(OutputFormat):
def out_typedef(self, fname, name, args):
module = self.modulename
purpose = args.get('purpose')
+ out_name = self.arg_name(args, name)
- self.data += f'.TH "{module}" 9 "{name}" "{self.man_date}" "API Manual" LINUX' + "\n"
+ self.data += f'.TH "{module}" 9 "{out_name}" "{self.man_date}" "API Manual" LINUX' + "\n"
self.data += ".SH NAME\n"
self.data += f"typedef {name} \\- {purpose}\n"
@@ -717,8 +790,9 @@ class ManFormat(OutputFormat):
module = self.modulename
purpose = args.get('purpose')
definition = args.get('definition')
+ out_name = self.arg_name(args, name)
- self.data += f'.TH "{module}" 9 "{args.type} {name}" "{self.man_date}" "API Manual" LINUX' + "\n"
+ self.data += f'.TH "{module}" 9 "{out_name}" "{self.man_date}" "API Manual" LINUX' + "\n"
self.data += ".SH NAME\n"
self.data += f"{args.type} {name} \\- {purpose}\n"
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 18/19] scripts: kdoc_parser.py: warn about Python version only once
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (16 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 17/19] tools: kernel-doc: add a see also section at man pages Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-04 7:33 ` [PATCH v4 19/19] tools/docs: sphinx-* break documentation bulds on openSUSE Mauro Carvalho Chehab
2025-09-05 16:07 ` [PATCH v4 00/19] Split sphinx call logic from docs Makefile Jonathan Corbet
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
When running kernel-doc over multiple documents, it emits
one error message per file with is not what we want:
$ python3.6 scripts/kernel-doc.py . --none
...
Warning: ./include/trace/events/swiotlb.h:0 Python 3.7 or later is required for correct results
Warning: ./include/trace/events/iommu.h:0 Python 3.7 or later is required for correct results
Warning: ./include/trace/events/sock.h:0 Python 3.7 or later is required for correct results
...
Change the logic to warn it only once at the library:
$ python3.6 scripts/kernel-doc.py . --none
Warning: Python 3.7 or later is required for correct results
Warning: ./include/cxl/features.h:0 Python 3.7 or later is required for correct results
When running from command line, it warns twice, but that sounds
ok.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/lib/kdoc/kdoc_parser.py | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index a560546c1867..574972e1f741 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -314,6 +314,7 @@ class KernelEntry:
self.section = SECTION_DEFAULT
self._contents = []
+python_warning = False
class KernelDoc:
"""
@@ -347,9 +348,13 @@ class KernelDoc:
# We need Python 3.7 for its "dicts remember the insertion
# order" guarantee
#
- if sys.version_info.major == 3 and sys.version_info.minor < 7:
+ global python_warning
+ if (not python_warning and
+ sys.version_info.major == 3 and sys.version_info.minor < 7):
+
self.emit_msg(0,
'Python 3.7 or later is required for correct results')
+ python_warning = True
def emit_msg(self, ln, msg, warning=True):
"""Emit a message"""
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* [PATCH v4 19/19] tools/docs: sphinx-* break documentation bulds on openSUSE
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (17 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 18/19] scripts: kdoc_parser.py: warn about Python version only once Mauro Carvalho Chehab
@ 2025-09-04 7:33 ` Mauro Carvalho Chehab
2025-09-05 16:07 ` [PATCH v4 00/19] Split sphinx call logic from docs Makefile Jonathan Corbet
19 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-04 7:33 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Andreas Hindborg, Benno Lossin, Boqun Feng,
Danilo Krummrich, Gary Guo, Mauro Carvalho Chehab, Miguel Ojeda,
Trevor Gross, linux-kernel, rust-for-linux
Before this patch, building htmldocs on opensuseLEAP works
fine:
# make htmldocs
Available Python versions:
/usr/bin/python3.11
Python 3.6.15 not supported. Changing to /usr/bin/python3.11
Python 3.6.15 not supported. Changing to /usr/bin/python3.11
Using alabaster theme
Using Python kernel-doc
...
As the logic detects that Python 3.6 is too old and recommends
intalling python311-Sphinx. If installed, documentation builds
work like a charm.
Yet, some develpers complained that running python3.11 instead
of python3 should not happen. So, let's break the build to make
them happier:
$ make htmldocs
Python 3.6.15 not supported. Bailing out
You could run, instead:
/usr/bin/python3.11 tools/docs/sphinx-build-wrapper htmldocs \
--sphinxdirs=. --conf=conf.py --builddir=Documentation/output --theme= --css= \
--paper=
Python 3.6.15 not supported. Bailing out
make[2]: *** [Documentation/Makefile:76: htmldocs] Error 1
make[1]: *** [Makefile:1806: htmldocs] Error 2
make: *** [Makefile:248: __sub-make] Error 2
It should be noticed that:
1. after this change, sphinx-pre-install needs to be called
by hand:
$ /usr/bin/python3.11 tools/docs/sphinx-pre-install
Detected OS: openSUSE Leap 15.6.
Sphinx version: 7.2.6
All optional dependencies are met.
Needed package dependencies are met.
2. sphinx-build-wrapper will auto-detect python3.11 and
suggest a way to build the docs using the parameters passed
via make variables. In this specific example:
/usr/bin/python3.11 tools/docs/sphinx-build-wrapper htmldocs --sphinxdirs=. --conf=conf.py --theme= --css= --paper=
3. As this needs to be executed outside docs Makefile, it won't run
the validation check scripts nor build Rust documentation if
enabled, as the extra scripts are part of the docs Makefile.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/lib/python_version.py | 28 ++++++++++++++++++++++++----
tools/docs/sphinx-build-wrapper | 3 ++-
tools/docs/sphinx-pre-install | 3 ++-
3 files changed, 28 insertions(+), 6 deletions(-)
diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_version.py
index a9fda2470a26..4fde1b882164 100644
--- a/tools/docs/lib/python_version.py
+++ b/tools/docs/lib/python_version.py
@@ -20,9 +20,11 @@ Python version if present.
import os
import re
import subprocess
+import shlex
import sys
from glob import glob
+from textwrap import indent
class PythonVersion:
"""
@@ -44,6 +46,25 @@ class PythonVersion:
"""Returns a version tuple as major.minor.patch"""
return ".".join([str(x) for x in version])
+ @staticmethod
+ def cmd_print(cmd, max_len=80):
+ cmd_line = []
+
+ for w in cmd:
+ w = shlex.quote(w)
+
+ if cmd_line:
+ if not max_len or len(cmd_line[-1]) + len(w) < max_len:
+ cmd_line[-1] += " " + w
+ continue
+ else:
+ cmd_line[-1] += " \\"
+ cmd_line.append(w)
+ else:
+ cmd_line.append(w)
+
+ return "\n ".join(cmd_line)
+
def __str__(self):
"""Returns a version tuple as major.minor.patch from self.version"""
return self.ver_str(self.version)
@@ -130,14 +151,13 @@ class PythonVersion:
else:
new_python_cmd = None
- if show_alternatives:
+ if show_alternatives and available_versions:
print("You could run, instead:")
for _, cmd in available_versions:
args = [cmd, script_path] + sys.argv[1:]
- cmd_str = " ".join(args)
- print(f" {cmd_str}")
- print()
+ cmd_str = indent(PythonVersion.cmd_print(args), " ")
+ print(f"{cmd_str}\n")
if bail_out:
msg = f"Python {python_ver} not supported. Bailing out"
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index 183717eb6d87..897c3512d1c7 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -725,7 +725,8 @@ def main():
args = parser.parse_args()
- PythonVersion.check_python(MIN_PYTHON_VERSION)
+ PythonVersion.check_python(MIN_PYTHON_VERSION, show_alternatives=True,
+ bail_out=True)
builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
verbose=args.verbose, n_jobs=args.jobs,
diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install
index 663d4e2a3f57..698989584b6a 100755
--- a/tools/docs/sphinx-pre-install
+++ b/tools/docs/sphinx-pre-install
@@ -1531,7 +1531,8 @@ def main():
checker = SphinxDependencyChecker(args)
- PythonVersion.check_python(MIN_PYTHON_VERSION)
+ PythonVersion.check_python(MIN_PYTHON_VERSION,
+ bail_out=True, success_on_error=True)
checker.check_needs()
# Call main if not used as module
--
2.51.0
^ permalink raw reply related [flat|nested] 67+ messages in thread* Re: [PATCH v4 00/19] Split sphinx call logic from docs Makefile
2025-09-04 7:33 [PATCH v4 00/19] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (18 preceding siblings ...)
2025-09-04 7:33 ` [PATCH v4 19/19] tools/docs: sphinx-* break documentation bulds on openSUSE Mauro Carvalho Chehab
@ 2025-09-05 16:07 ` Jonathan Corbet
2025-09-06 9:40 ` Mauro Carvalho Chehab
19 siblings, 1 reply; 67+ messages in thread
From: Jonathan Corbet @ 2025-09-05 16:07 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel,
Matthew Wilcox, linux-doc
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> This series does a major cleanup at docs Makefile by moving the
> actual doc build logic to a helper script (scripts/sphinx-build-wrapper).
>
> Such script was written in a way that it can be called either
> directly or via a makefile. When running via makefile, it will
> use GNU jobserver to ensure that, when sphinx-build is
> called, the number of jobs will match at most what it is
> specified by the "-j" parameter.
I will try to make another pass over this stuff later today. I would
really appreciate some more eyes on it, though, and perhaps even some
reports of testing. This is a significant change, and the presence of
surprised would not be ... surprising ...
Thanks,
jon
^ permalink raw reply [flat|nested] 67+ messages in thread* Re: [PATCH v4 00/19] Split sphinx call logic from docs Makefile
2025-09-05 16:07 ` [PATCH v4 00/19] Split sphinx call logic from docs Makefile Jonathan Corbet
@ 2025-09-06 9:40 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 67+ messages in thread
From: Mauro Carvalho Chehab @ 2025-09-06 9:40 UTC (permalink / raw)
To: Jonathan Corbet; +Cc: Linux Doc Mailing List, linux-kernel, Matthew Wilcox
Em Fri, 05 Sep 2025 10:07:51 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
> > This series does a major cleanup at docs Makefile by moving the
> > actual doc build logic to a helper script (scripts/sphinx-build-wrapper).
> >
> > Such script was written in a way that it can be called either
> > directly or via a makefile. When running via makefile, it will
> > use GNU jobserver to ensure that, when sphinx-build is
> > called, the number of jobs will match at most what it is
> > specified by the "-j" parameter.
>
> I will try to make another pass over this stuff later today. I would
> really appreciate some more eyes on it, though, and perhaps even some
> reports of testing. This is a significant change, and the presence of
> surprised would not be ... surprising ...
Take your time.
Yeah, tests are very welcomed. The change itself is not significant
in the sense that we just moved a complex logic with lots of magic
from Makefile to Python (*) without changing the implementation, but
yeah, it requires testing. From my side, I've been testing cleandocs,
htmldocs and pdfdocs for 3-4 weeks now on over ~20 different distros,
all without O=, but didn't make any burn-in test for the other targets.
I've got (and fixed) some corner cases, but I won't doubt that other
corner cases might be there.
(*) It actually solved one issue: with current implementation, at least for me,
using V=1 doesn't show the sphinx-build command line, as this is hidden
inside the complex makefile foreach macro:
loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
cmd_sphinx = \
PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \
$(PYTHON3) $(srctree)/scripts/jobserver-exec \
$(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
$(SPHINXBUILD) \
-b $2 \
-c $(abspath $(src)) \
-d $(abspath $(BUILDDIR)/.doctrees/$3) \
-D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
$(ALLSPHINXOPTS) \
$(abspath $(src)/$5) \
$(abspath $(BUILDDIR)/$3/$4) && \
if [ "x$(DOCS_CSS)" != "x" ]; then \
cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(DOCS_CSS)) $(BUILDDIR)/$3/_static/; \
fi
htmldocs:
...
@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
IMHO, this is problematic, as it makes harder to debug corner
cases. While doing this changeset, I actually had to add an echo
line to show the command line, to ensure that sphinx-build was
called with the same arguments.
On a side note, the above macro is complex and hard to maintain.
One of the reasons why the Python script is bigger is that it
has one statement per line instead of trying to do lots of
calls inside a single statement like above (there, cmd_sphinx
has 8 function calls; htmldocs foreach line has 3).
The core of cmd_sphinx grew from 14 LOC on Makefile to 64 lines
in Python, not counting:
- comments
- CSS logic
- path handling logic
- jobserver logic (in Makefile, this is a single "+" character,
at the foreach line)
I remember I had to touch on the foreach/call logic there a couple
of times in the past. It is not the easiest thing to do.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 67+ messages in thread