* [PATCH 01/11] scripts/jobserver-exec: move the code to a class
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 02/11] scripts/jobserver-exec: move its class to the lib directory Mauro Carvalho Chehab
` (9 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Convert the code inside jobserver-exec to a class and
properly document it.
Using a class allows reusing the jobserver logic on other
scripts.
While the main code remains unchanged, being compatible with
Python 2.6 and 3.0+, its coding style now follows a more
modern standard, having tabs replaced by a 4-spaces
indent, passing autopep8, black and pylint.
The code now allows allows using a pythonic way to
enter/exit a python code, e.g. it now supports:
with JobserverExec() as jobserver:
jobserver.run(sys.argv[1:])
With the new code, the __exit__() function should ensure
that the jobserver slot will be closed at the end, even if
something bad happens somewhere.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/jobserver-exec | 218 ++++++++++++++++++++++++++++-------------
1 file changed, 151 insertions(+), 67 deletions(-)
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 7eca035472d3..b386b1a845de 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -1,77 +1,161 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
#
+# pylint: disable=C0103,C0209
+#
# This determines how many parallel tasks "make" is expecting, as it is
# not exposed via an special variables, reserves them all, runs a subprocess
# with PARALLELISM environment variable set, and releases the jobs back again.
#
# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
-from __future__ import print_function
-import os, sys, errno
+
+"""
+Interacts with the POSIX jobserver during the Kernel build time.
+
+A "normal" jobserver task, like the one initiated by a make subrocess would do:
+
+ - open read/write file descriptors to communicate with the job server;
+ - ask for one slot by calling:
+ claim = os.read(reader, 1)
+ - when the job finshes, call:
+ os.write(writer, b"+") # os.write(writer, claim)
+
+Here, the goal is different: This script aims to get the remaining number
+of slots available, using all of them to run a command which handle tasks in
+parallel. To to that, it has a loop that ends only after there are no
+slots left. It then increments the number by one, in order to allow a
+call equivalent to make -j$((claim+1)), e.g. having a parent make creating
+$claim child to do the actual work.
+
+The end goal here is to keep the total number of build tasks under the
+limit established by the initial make -j$n_proc call.
+"""
+
+import errno
+import os
import subprocess
+import sys
-# Extract and prepare jobserver file descriptors from environment.
-claim = 0
-jobs = b""
-try:
- # Fetch the make environment options.
- flags = os.environ['MAKEFLAGS']
-
- # Look for "--jobserver=R,W"
- # Note that GNU Make has used --jobserver-fds and --jobserver-auth
- # so this handles all of them.
- opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
-
- # Parse out R,W file descriptor numbers and set them nonblocking.
- # If the MAKEFLAGS variable contains multiple instances of the
- # --jobserver-auth= option, the last one is relevant.
- fds = opts[-1].split("=", 1)[1]
-
- # Starting with GNU Make 4.4, named pipes are used for reader and writer.
- # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
- _, _, path = fds.partition('fifo:')
-
- if path:
- reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
- writer = os.open(path, os.O_WRONLY)
- else:
- reader, writer = [int(x) for x in fds.split(",", 1)]
- # Open a private copy of reader to avoid setting nonblocking
- # on an unexpecting process with the same reader fd.
- reader = os.open("/proc/self/fd/%d" % (reader),
- os.O_RDONLY | os.O_NONBLOCK)
-
- # Read out as many jobserver slots as possible.
- while True:
- try:
- slot = os.read(reader, 8)
- jobs += slot
- except (OSError, IOError) as e:
- if e.errno == errno.EWOULDBLOCK:
- # Stop at the end of the jobserver queue.
- break
- # If something went wrong, give back the jobs.
- if len(jobs):
- os.write(writer, jobs)
- raise e
- # Add a bump for our caller's reserveration, since we're just going
- # to sit here blocked on our child.
- claim = len(jobs) + 1
-except (KeyError, IndexError, ValueError, OSError, IOError) as e:
- # Any missing environment strings or bad fds should result in just
- # not being parallel.
- pass
-
-# We can only claim parallelism if there was a jobserver (i.e. a top-level
-# "-jN" argument) and there were no other failures. Otherwise leave out the
-# environment variable and let the child figure out what is best.
-if claim > 0:
- os.environ['PARALLELISM'] = '%d' % (claim)
-
-rc = subprocess.call(sys.argv[1:])
-
-# Return all the reserved slots.
-if len(jobs):
- os.write(writer, jobs)
-
-sys.exit(rc)
+
+class JobserverExec:
+ """
+ Claim all slots from make using POSIX Jobserver.
+
+ The main methods here are:
+ - open(): reserves all slots;
+ - close(): method returns all used slots back to make;
+ - run(): executes a command setting PARALLELISM=<available slots jobs + 1>
+ """
+
+ def __init__(self):
+ """Initialize internal vars"""
+ self.claim = 0
+ self.jobs = b""
+ self.reader = None
+ self.writer = None
+ self.is_open = False
+
+ def open(self):
+ """Reserve all available slots to be claimed later on"""
+
+ if self.is_open:
+ return
+
+ try:
+ # Fetch the make environment options.
+ flags = os.environ["MAKEFLAGS"]
+ # Look for "--jobserver=R,W"
+ # Note that GNU Make has used --jobserver-fds and --jobserver-auth
+ # so this handles all of them.
+ opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
+ # Parse out R,W file descriptor numbers and set them nonblocking.
+ # If the MAKEFLAGS variable contains multiple instances of the
+ # --jobserver-auth= option, the last one is relevant.
+ fds = opts[-1].split("=", 1)[1]
+
+ # Starting with GNU Make 4.4, named pipes are used for reader
+ # and writer.
+ # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
+ _, _, path = fds.partition("fifo:")
+
+ if path:
+ self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
+ self.writer = os.open(path, os.O_WRONLY)
+ else:
+ self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
+ # Open a private copy of reader to avoid setting nonblocking
+ # on an unexpecting process with the same reader fd.
+ self.reader = os.open("/proc/self/fd/%d" % (self.reader),
+ os.O_RDONLY | os.O_NONBLOCK)
+
+ # Read out as many jobserver slots as possible
+ while True:
+ try:
+ slot = os.read(self.reader, 8)
+ self.jobs += slot
+ except (OSError, IOError) as e:
+ if e.errno == errno.EWOULDBLOCK:
+ # Stop at the end of the jobserver queue.
+ break
+ # If something went wrong, give back the jobs.
+ if self.jobs:
+ os.write(self.writer, self.jobs)
+ raise e
+
+ # Add a bump for our caller's reserveration, since we're just going
+ # to sit here blocked on our child.
+ self.claim = len(self.jobs) + 1
+
+ except (KeyError, IndexError, ValueError, OSError, IOError):
+ # Any missing environment strings or bad fds should result in just
+ # not being parallel.
+ self.claim = None
+
+ self.is_open = True
+
+ def close(self):
+ """Return all reserved slots to Jobserver"""
+
+ if not self.is_open:
+ return
+
+ # Return all the reserved slots.
+ if len(self.jobs):
+ os.write(self.writer, self.jobs)
+
+ self.is_open = False
+
+ def __enter__(self):
+ self.open()
+ return self
+
+ def __exit__(self, exc_type, exc_value, exc_traceback):
+ self.close()
+
+ def run(self, cmd):
+ """
+ Run a command setting PARALLELISM env variable to the number of
+ available job slots (claim) + 1, e.g. it will reserve claim slots
+ to do the actual build work, plus one to monitor its childs.
+ """
+ self.open() # Ensure that self.claim is set
+
+ # We can only claim parallelism if there was a jobserver (i.e. a
+ # top-level "-jN" argument) and there were no other failures. Otherwise
+ # leave out the environment variable and let the child figure out what
+ # is best.
+ if self.claim:
+ os.environ["PARALLELISM"] = str(self.claim)
+
+ return subprocess.call(cmd)
+
+
+def main():
+ """Main program"""
+ with JobserverExec() as jobserver:
+ jobserver.run(sys.argv[1:])
+
+
+if __name__ == "__main__":
+ main()
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 02/11] scripts/jobserver-exec: move its class to the lib directory
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 01/11] scripts/jobserver-exec: move the code to a class Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 03/11] scripts/jobserver-exec: add a help message Mauro Carvalho Chehab
` (8 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
To make it easier to be re-used, move the JobserverExec class
to the library directory.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/jobserver-exec | 152 +++------------------------------------
scripts/lib/jobserver.py | 149 ++++++++++++++++++++++++++++++++++++++
2 files changed, 160 insertions(+), 141 deletions(-)
create mode 100755 scripts/lib/jobserver.py
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index b386b1a845de..40a0f0058733 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -1,155 +1,25 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
-#
-# pylint: disable=C0103,C0209
-#
-# This determines how many parallel tasks "make" is expecting, as it is
-# not exposed via an special variables, reserves them all, runs a subprocess
-# with PARALLELISM environment variable set, and releases the jobs back again.
-#
-# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
-"""
-Interacts with the POSIX jobserver during the Kernel build time.
-
-A "normal" jobserver task, like the one initiated by a make subrocess would do:
-
- - open read/write file descriptors to communicate with the job server;
- - ask for one slot by calling:
- claim = os.read(reader, 1)
- - when the job finshes, call:
- os.write(writer, b"+") # os.write(writer, claim)
-
-Here, the goal is different: This script aims to get the remaining number
-of slots available, using all of them to run a command which handle tasks in
-parallel. To to that, it has a loop that ends only after there are no
-slots left. It then increments the number by one, in order to allow a
-call equivalent to make -j$((claim+1)), e.g. having a parent make creating
-$claim child to do the actual work.
-
-The end goal here is to keep the total number of build tasks under the
-limit established by the initial make -j$n_proc call.
-"""
-
-import errno
import os
-import subprocess
import sys
+LIB_DIR = "lib"
+SRC_DIR = os.path.dirname(os.path.realpath(__file__))
-class JobserverExec:
- """
- Claim all slots from make using POSIX Jobserver.
+sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
- The main methods here are:
- - open(): reserves all slots;
- - close(): method returns all used slots back to make;
- - run(): executes a command setting PARALLELISM=<available slots jobs + 1>
- """
+from jobserver import JobserverExec # pylint: disable=C0415
- def __init__(self):
- """Initialize internal vars"""
- self.claim = 0
- self.jobs = b""
- self.reader = None
- self.writer = None
- self.is_open = False
- def open(self):
- """Reserve all available slots to be claimed later on"""
-
- if self.is_open:
- return
-
- try:
- # Fetch the make environment options.
- flags = os.environ["MAKEFLAGS"]
- # Look for "--jobserver=R,W"
- # Note that GNU Make has used --jobserver-fds and --jobserver-auth
- # so this handles all of them.
- opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
-
- # Parse out R,W file descriptor numbers and set them nonblocking.
- # If the MAKEFLAGS variable contains multiple instances of the
- # --jobserver-auth= option, the last one is relevant.
- fds = opts[-1].split("=", 1)[1]
-
- # Starting with GNU Make 4.4, named pipes are used for reader
- # and writer.
- # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
- _, _, path = fds.partition("fifo:")
-
- if path:
- self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
- self.writer = os.open(path, os.O_WRONLY)
- else:
- self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
- # Open a private copy of reader to avoid setting nonblocking
- # on an unexpecting process with the same reader fd.
- self.reader = os.open("/proc/self/fd/%d" % (self.reader),
- os.O_RDONLY | os.O_NONBLOCK)
-
- # Read out as many jobserver slots as possible
- while True:
- try:
- slot = os.read(self.reader, 8)
- self.jobs += slot
- except (OSError, IOError) as e:
- if e.errno == errno.EWOULDBLOCK:
- # Stop at the end of the jobserver queue.
- break
- # If something went wrong, give back the jobs.
- if self.jobs:
- os.write(self.writer, self.jobs)
- raise e
-
- # Add a bump for our caller's reserveration, since we're just going
- # to sit here blocked on our child.
- self.claim = len(self.jobs) + 1
-
- except (KeyError, IndexError, ValueError, OSError, IOError):
- # Any missing environment strings or bad fds should result in just
- # not being parallel.
- self.claim = None
-
- self.is_open = True
-
- def close(self):
- """Return all reserved slots to Jobserver"""
-
- if not self.is_open:
- return
-
- # Return all the reserved slots.
- if len(self.jobs):
- os.write(self.writer, self.jobs)
-
- self.is_open = False
-
- def __enter__(self):
- self.open()
- return self
-
- def __exit__(self, exc_type, exc_value, exc_traceback):
- self.close()
-
- def run(self, cmd):
- """
- Run a command setting PARALLELISM env variable to the number of
- available job slots (claim) + 1, e.g. it will reserve claim slots
- to do the actual build work, plus one to monitor its childs.
- """
- self.open() # Ensure that self.claim is set
-
- # We can only claim parallelism if there was a jobserver (i.e. a
- # top-level "-jN" argument) and there were no other failures. Otherwise
- # leave out the environment variable and let the child figure out what
- # is best.
- if self.claim:
- os.environ["PARALLELISM"] = str(self.claim)
-
- return subprocess.call(cmd)
+"""
+Determines how many parallel tasks "make" is expecting, as it is
+not exposed via an special variables, reserves them all, runs a subprocess
+with PARALLELISM environment variable set, and releases the jobs back again.
+See:
+ https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+"""
def main():
"""Main program"""
diff --git a/scripts/lib/jobserver.py b/scripts/lib/jobserver.py
new file mode 100755
index 000000000000..98d8b0ff0c89
--- /dev/null
+++ b/scripts/lib/jobserver.py
@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0+
+#
+# pylint: disable=C0103,C0209
+#
+#
+
+"""
+Interacts with the POSIX jobserver during the Kernel build time.
+
+A "normal" jobserver task, like the one initiated by a make subrocess would do:
+
+ - open read/write file descriptors to communicate with the job server;
+ - ask for one slot by calling:
+ claim = os.read(reader, 1)
+ - when the job finshes, call:
+ os.write(writer, b"+") # os.write(writer, claim)
+
+Here, the goal is different: This script aims to get the remaining number
+of slots available, using all of them to run a command which handle tasks in
+parallel. To to that, it has a loop that ends only after there are no
+slots left. It then increments the number by one, in order to allow a
+call equivalent to make -j$((claim+1)), e.g. having a parent make creating
+$claim child to do the actual work.
+
+The end goal here is to keep the total number of build tasks under the
+limit established by the initial make -j$n_proc call.
+
+See:
+ https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+"""
+
+import errno
+import os
+import subprocess
+import sys
+
+class JobserverExec:
+ """
+ Claim all slots from make using POSIX Jobserver.
+
+ The main methods here are:
+ - open(): reserves all slots;
+ - close(): method returns all used slots back to make;
+ - run(): executes a command setting PARALLELISM=<available slots jobs + 1>
+ """
+
+ def __init__(self):
+ """Initialize internal vars"""
+ self.claim = 0
+ self.jobs = b""
+ self.reader = None
+ self.writer = None
+ self.is_open = False
+
+ def open(self):
+ """Reserve all available slots to be claimed later on"""
+
+ if self.is_open:
+ return
+
+ try:
+ # Fetch the make environment options.
+ flags = os.environ["MAKEFLAGS"]
+ # Look for "--jobserver=R,W"
+ # Note that GNU Make has used --jobserver-fds and --jobserver-auth
+ # so this handles all of them.
+ opts = [x for x in flags.split(" ") if x.startswith("--jobserver")]
+
+ # Parse out R,W file descriptor numbers and set them nonblocking.
+ # If the MAKEFLAGS variable contains multiple instances of the
+ # --jobserver-auth= option, the last one is relevant.
+ fds = opts[-1].split("=", 1)[1]
+
+ # Starting with GNU Make 4.4, named pipes are used for reader
+ # and writer.
+ # Example argument: --jobserver-auth=fifo:/tmp/GMfifo8134
+ _, _, path = fds.partition("fifo:")
+
+ if path:
+ self.reader = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
+ self.writer = os.open(path, os.O_WRONLY)
+ else:
+ self.reader, self.writer = [int(x) for x in fds.split(",", 1)]
+ # Open a private copy of reader to avoid setting nonblocking
+ # on an unexpecting process with the same reader fd.
+ self.reader = os.open("/proc/self/fd/%d" % (self.reader),
+ os.O_RDONLY | os.O_NONBLOCK)
+
+ # Read out as many jobserver slots as possible
+ while True:
+ try:
+ slot = os.read(self.reader, 8)
+ self.jobs += slot
+ except (OSError, IOError) as e:
+ if e.errno == errno.EWOULDBLOCK:
+ # Stop at the end of the jobserver queue.
+ break
+ # If something went wrong, give back the jobs.
+ if self.jobs:
+ os.write(self.writer, self.jobs)
+ raise e
+
+ # Add a bump for our caller's reserveration, since we're just going
+ # to sit here blocked on our child.
+ self.claim = len(self.jobs) + 1
+
+ except (KeyError, IndexError, ValueError, OSError, IOError):
+ # Any missing environment strings or bad fds should result in just
+ # not being parallel.
+ self.claim = None
+
+ self.is_open = True
+
+ def close(self):
+ """Return all reserved slots to Jobserver"""
+
+ if not self.is_open:
+ return
+
+ # Return all the reserved slots.
+ if len(self.jobs):
+ os.write(self.writer, self.jobs)
+
+ self.is_open = False
+
+ def __enter__(self):
+ self.open()
+ return self
+
+ def __exit__(self, exc_type, exc_value, exc_traceback):
+ self.close()
+
+ def run(self, cmd, *args, **pwargs):
+ """
+ Run a command setting PARALLELISM env variable to the number of
+ available job slots (claim) + 1, e.g. it will reserve claim slots
+ to do the actual build work, plus one to monitor its childs.
+ """
+ self.open() # Ensure that self.claim is set
+
+ # We can only claim parallelism if there was a jobserver (i.e. a
+ # top-level "-jN" argument) and there were no other failures. Otherwise
+ # leave out the environment variable and let the child figure out what
+ # is best.
+ if self.claim:
+ os.environ["PARALLELISM"] = str(self.claim)
+
+ return subprocess.call(cmd, *args, **pwargs)
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 03/11] scripts/jobserver-exec: add a help message
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 01/11] scripts/jobserver-exec: move the code to a class Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 02/11] scripts/jobserver-exec: move its class to the lib directory Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
` (7 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Currently, calling it without an argument shows an ugly error
message. Instead, print a message using pythondoc as description.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/jobserver-exec | 22 +++++++++++++---------
1 file changed, 13 insertions(+), 9 deletions(-)
diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 40a0f0058733..ae23afd344ec 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -1,6 +1,15 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: GPL-2.0+
+"""
+Determines how many parallel tasks "make" is expecting, as it is
+not exposed via any special variables, reserves them all, runs a subprocess
+with PARALLELISM environment variable set, and releases the jobs back again.
+
+See:
+ https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
+"""
+
import os
import sys
@@ -12,17 +21,12 @@ sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
from jobserver import JobserverExec # pylint: disable=C0415
-"""
-Determines how many parallel tasks "make" is expecting, as it is
-not exposed via an special variables, reserves them all, runs a subprocess
-with PARALLELISM environment variable set, and releases the jobs back again.
-
-See:
- https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
-"""
-
def main():
"""Main program"""
+ if len(sys.argv) < 2:
+ name = os.path.basename(__file__)
+ sys.exit("usage: " + name +" command [args ...]\n" + __doc__)
+
with JobserverExec() as jobserver:
jobserver.run(sys.argv[1:])
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (2 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 03/11] scripts/jobserver-exec: add a help message Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-16 1:16 ` Akira Yokosawa
` (2 more replies)
2025-08-15 11:50 ` [PATCH 05/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper Mauro Carvalho Chehab
` (6 subsequent siblings)
10 siblings, 3 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron,
Mauro Carvalho Chehab, Alex Gaynor, Alice Ryhl, Andreas Hindborg,
Benno Lossin, Boqun Feng, Danilo Krummrich, Gary Guo,
Miguel Ojeda, Trevor Gross, linux-kernel, rust-for-linux
There are too much magic inside docs Makefile to properly run
sphinx-build. Create an ancillary script that contains all
kernel-related sphinx-build call logic currently at Makefile.
Such script is designed to work both as an standalone command
and as part of a Makefile. As such, it properly handles POSIX
jobserver used by GNU make.
It should be noticed that, when running the script alone,
it will only take care of sphinx-build and cleandocs target.
As such:
- it won't run "make rustdoc";
- no extra checks.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
.pylintrc | 2 +-
scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++
2 files changed, 628 insertions(+), 1 deletion(-)
create mode 100755 scripts/sphinx-build-wrapper
diff --git a/.pylintrc b/.pylintrc
index 30b8ae1659f8..f1d21379254b 100644
--- a/.pylintrc
+++ b/.pylintrc
@@ -1,2 +1,2 @@
[MASTER]
-init-hook='import sys; sys.path += ["scripts/lib/kdoc", "scripts/lib/abi"]'
+init-hook='import sys; sys.path += ["scripts/lib", "scripts/lib/kdoc", "scripts/lib/abi"]'
diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper
new file mode 100755
index 000000000000..5c728956b53c
--- /dev/null
+++ b/scripts/sphinx-build-wrapper
@@ -0,0 +1,627 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2025 Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
+#
+# pylint: disable=R0902, R0912, R0913, R0914, R0915, R0917, C0103
+#
+# Converted from docs Makefile and parallel-wrapper.sh, both under
+# GPLv2, copyrighted since 2008 by the following authors:
+#
+# Akira Yokosawa <akiyks@gmail.com>
+# Arnd Bergmann <arnd@arndb.de>
+# Breno Leitao <leitao@debian.org>
+# Carlos Bilbao <carlos.bilbao@amd.com>
+# Dave Young <dyoung@redhat.com>
+# Donald Hunter <donald.hunter@gmail.com>
+# Geert Uytterhoeven <geert+renesas@glider.be>
+# Jani Nikula <jani.nikula@intel.com>
+# Jan Stancek <jstancek@redhat.com>
+# Jonathan Corbet <corbet@lwn.net>
+# Joshua Clayton <stillcompiling@gmail.com>
+# Kees Cook <keescook@chromium.org>
+# Linus Torvalds <torvalds@linux-foundation.org>
+# Magnus Damm <damm+renesas@opensource.se>
+# Masahiro Yamada <masahiroy@kernel.org>
+# Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
+# Maxim Cournoyer <maxim.cournoyer@gmail.com>
+# Peter Foley <pefoley2@pefoley.com>
+# Randy Dunlap <rdunlap@infradead.org>
+# Rob Herring <robh@kernel.org>
+# Shuah Khan <shuahkh@osg.samsung.com>
+# Thorsten Blum <thorsten.blum@toblux.com>
+# Tomas Winkler <tomas.winkler@intel.com>
+
+
+"""
+Sphinx build wrapper that handles Kernel-specific business rules:
+
+- it gets the Kernel build environment vars;
+- it determines what's the best parallelism;
+- it handles SPHINXDIRS
+
+This tool ensures that MIN_PYTHON_VERSION is satisfied. If version is
+below that, it seeks for a new Python version. If found, it re-runs using
+the newer version.
+"""
+
+import argparse
+import os
+import re
+import shlex
+import shutil
+import subprocess
+import sys
+
+from glob import glob
+
+LIB_DIR = "lib"
+SRC_DIR = os.path.dirname(os.path.realpath(__file__))
+
+sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR))
+
+from jobserver import JobserverExec # pylint: disable=C0413
+
+
+def parse_version(version):
+ """Convert a major.minor.patch version into a tuple"""
+ return tuple(int(x) for x in version.split("."))
+
+def ver_str(version):
+ """Returns a version tuple as major.minor.patch"""
+
+ return ".".join([str(x) for x in version])
+
+# Minimal supported Python version needed by Sphinx and its extensions
+MIN_PYTHON_VERSION = parse_version("3.7")
+
+# Default value for --venv parameter
+VENV_DEFAULT = "sphinx_latest"
+
+# List of make targets and its corresponding builder and output directory
+TARGETS = {
+ "cleandocs": {
+ "builder": "clean",
+ },
+ "htmldocs": {
+ "builder": "html",
+ },
+ "epubdocs": {
+ "builder": "epub",
+ "out_dir": "epub",
+ },
+ "texinfodocs": {
+ "builder": "texinfo",
+ "out_dir": "texinfo",
+ },
+ "infodocs": {
+ "builder": "texinfo",
+ "out_dir": "texinfo",
+ },
+ "latexdocs": {
+ "builder": "latex",
+ "out_dir": "latex",
+ },
+ "pdfdocs": {
+ "builder": "latex",
+ "out_dir": "latex",
+ },
+ "xmldocs": {
+ "builder": "xml",
+ "out_dir": "xml",
+ },
+ "linkcheckdocs": {
+ "builder": "linkcheck"
+ },
+}
+
+# Paper sizes. An empty value will pick the default
+PAPER = ["", "a4", "letter"]
+
+class SphinxBuilder:
+ """
+ Handles a sphinx-build target, adding needed arguments to build
+ with the Kernel.
+ """
+
+ def is_rust_enabled(self):
+ """Check if rust is enabled at .config"""
+ config_path = os.path.join(self.srctree, ".config")
+ if os.path.isfile(config_path):
+ with open(config_path, "r", encoding="utf-8") as f:
+ return "CONFIG_RUST=y" in f.read()
+ return False
+
+ def get_path(self, path, abs_path=False):
+ """
+ Ancillary routine to handle patches the right way, as shell does.
+
+ It first expands "~" and "~user". Then, if patch is not absolute,
+ join self.srctree. Finally, if requested, convert to abspath.
+ """
+
+ path = os.path.expanduser(path)
+ if not path.startswith("/"):
+ path = os.path.join(self.srctree, path)
+
+ if abs_path:
+ return os.path.abspath(path)
+
+ return path
+
+ def __init__(self, venv=None, verbose=False):
+ """Initialize internal variables"""
+ self.venv = venv
+ self.verbose = verbose
+
+ # Normal variables passed from Kernel's makefile
+ self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
+ self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
+ self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
+ self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+
+ # Source tree directory. This needs to be at os.environ, as
+ # Sphinx extensions and media uAPI makefile needs it
+ self.srctree = os.environ.get("srctree")
+ if not self.srctree:
+ self.srctree = "."
+ os.environ["srctree"] = self.srctree
+
+ # Now that we can expand srctree, get other directories as well
+ self.sphinxbuild = os.environ.get("SPHINXBUILD", "sphinx-build")
+ self.kerneldoc = self.get_path(os.environ.get("KERNELDOC",
+ "scripts/kernel-doc.py"))
+ self.obj = os.environ.get("obj", "Documentation")
+ self.builddir = self.get_path(os.path.join(self.obj, "output"),
+ abs_path=True)
+
+ # Media uAPI needs it
+ os.environ["BUILDDIR"] = self.builddir
+
+ # Detect if rust is enabled
+ self.config_rust = self.is_rust_enabled()
+
+ # Get directory locations for LaTeX build toolchain
+ self.pdflatex_cmd = shutil.which(self.pdflatex)
+ self.latexmk_cmd = shutil.which("latexmk")
+
+ self.env = os.environ.copy()
+
+ # If venv parameter is specified, run Sphinx from venv
+ if venv:
+ bin_dir = os.path.join(venv, "bin")
+ if os.path.isfile(os.path.join(bin_dir, "activate")):
+ # "activate" virtual env
+ self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
+ self.env["VIRTUAL_ENV"] = venv
+ if "PYTHONHOME" in self.env:
+ del self.env["PYTHONHOME"]
+ print(f"Setting venv to {venv}")
+ else:
+ sys.exit(f"Venv {venv} not found.")
+
+ def run_sphinx(self, sphinx_build, sphinx_args, *args, **pwargs):
+ """
+ Executes sphinx-build using current python3 command and setting
+ -j parameter if possible to run the build in parallel.
+ """
+
+ with JobserverExec() as jobserver:
+ if jobserver.claim:
+ parallelism = str(jobserver.claim)
+ else:
+ # As Sphinx has parallelism since version 1.7, we don't need
+ # any check here.
+ parallelism = "auto"
+
+ cmd = []
+
+ if self.venv:
+ cmd.append("python")
+ else:
+ cmd.append(sys.executable)
+
+ cmd.append(sphinx_build)
+
+ if parallelism:
+ cmd.append("-j" + parallelism)
+
+ cmd += sphinx_args
+
+ if self.verbose:
+ print(" ".join(cmd))
+
+ rc = subprocess.call(cmd, *args, **pwargs)
+
+ def handle_html(self, css, output_dir):
+ """
+ Extra steps for HTML and epub output.
+
+ For such targets, we need to ensure that CSS will be properly
+ copied to the output _static directory
+ """
+
+ if not css:
+ return
+
+ css = os.path.expanduser(css)
+ if not css.startswith("/"):
+ css = os.path.join(self.srctree, css)
+
+ static_dir = os.path.join(output_dir, "_static")
+ os.makedirs(static_dir, exist_ok=True)
+
+ try:
+ shutil.copy2(css, static_dir)
+ except (OSError, IOError) as e:
+ print(f"Warning: Failed to copy CSS: {e}", file=sys.stderr)
+
+ def handle_pdf(self, output_dirs):
+ """
+ Extra steps for PDF output.
+
+ As PDF is handled via a LaTeX output, after building the .tex file,
+ a new build is needed to create the PDF output from the latex
+ directory.
+ """
+ builds = {}
+ max_len = 0
+
+ for from_dir in output_dirs:
+ pdf_dir = os.path.join(from_dir, "../pdf")
+ os.makedirs(pdf_dir, exist_ok=True)
+
+ if self.latexmk_cmd:
+ latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
+ else:
+ latex_cmd = [self.pdflatex]
+
+ latex_cmd.extend(shlex.split(self.latexopts))
+
+ tex_suffix = ".tex"
+
+ # Process each .tex file
+ has_tex = False
+ build_failed = False
+ with os.scandir(from_dir) as it:
+ for entry in it:
+ if not entry.name.endswith(tex_suffix):
+ continue
+
+ name = entry.name[:-len(tex_suffix)]
+ has_tex = True
+
+ try:
+ subprocess.run(latex_cmd + [entry.path],
+ cwd=from_dir, check=True)
+ except subprocess.CalledProcessError:
+ # LaTeX PDF error code is almost useless: it returns
+ # error codes even when build succeeds but has warnings.
+ pass
+
+ # Instead of checking errors, let's do the next best thing:
+ # check if the PDF file was actually created.
+
+ pdf_name = name + ".pdf"
+ pdf_from = os.path.join(from_dir, pdf_name)
+ pdf_to = os.path.join(pdf_dir, pdf_name)
+
+ if os.path.exists(pdf_from):
+ os.rename(pdf_from, pdf_to)
+ builds[name] = os.path.relpath(pdf_to, self.builddir)
+ else:
+ builds[name] = "FAILED"
+ build_failed = True
+
+ name = entry.name.removesuffix(".tex")
+ max_len = max(max_len, len(name))
+
+ if not has_tex:
+ name = os.path.basename(from_dir)
+ max_len = max(max_len, len(name))
+ builds[name] = "FAILED (no .tex)"
+ build_failed = True
+
+ msg = "Summary"
+ msg += "\n" + "=" * len(msg)
+ print()
+ print(msg)
+
+ for pdf_name, pdf_file in builds.items():
+ print(f"{pdf_name:<{max_len}}: {pdf_file}")
+
+ print()
+
+ # return an error if a PDF file is missing
+
+ if build_failed:
+ sys.exit(f"PDF build failed: not all PDF files were created.")
+ else:
+ print("All PDF files were built.")
+
+ def handle_info(self, output_dirs):
+ """
+ Extra steps for Info output.
+
+ For texinfo generation, an additional make is needed from the
+ texinfo directory.
+ """
+
+ for output_dir in output_dirs:
+ try:
+ subprocess.run(["make", "info"], cwd=output_dir, check=True)
+ except subprocess.CalledProcessError as e:
+ sys.exit(f"Error generating info docs: {e}")
+
+ def get_make_media(self):
+ """
+ The media uAPI requires an additional Makefile target.
+ """
+
+ mediadir = f"{self.obj}/userspace-api/media"
+
+ make = os.environ.get("MAKE", "make")
+ build = os.environ.get("build", "-f $(srctree)/scripts/Makefile.build obj")
+
+ # Check if the script was started outside docs Makefile
+ if not os.environ.get("obj"):
+ mediadir = os.path.abspath(mediadir)
+
+ # the build makefile var contains macros that require expand
+ make_media = f"{make} {build}={mediadir}"
+ make_media = make_media.replace("$(", "${").replace(")", "}")
+ make_media = os.path.expandvars(make_media)
+
+ # As it also contains multiple arguments, use shlex to split it
+ return shlex.split(make_media)
+
+ def prepare_media(self, builder):
+ """
+ Run userspace-api/media Makefile.
+
+ The logic behind it are from the initial ports to Sphinx.
+ They're old and need to be replaced by a proper Sphinx extension.
+ While we don't do that, we need to explicitly call media Makefile
+ to build some files.
+ """
+
+ cmd = self.get_make_media() + [builder]
+
+ if self.verbose:
+ print(" ".join(cmd))
+
+ with JobserverExec() as jobserver:
+ rc = jobserver.run(cmd, env=self.env)
+
+ if rc:
+ cmd_str = " ".join(cmd)
+ sys.exit(f"Failed to run {cmd_str}")
+
+ def cleandocs(self, builder):
+
+ shutil.rmtree(self.builddir, ignore_errors=True)
+
+ self.prepare_media(builder)
+
+ def build(self, target, sphinxdirs=None, conf="conf.py",
+ theme=None, css=None, paper=None):
+ """
+ Build documentation using Sphinx. This is the core function of this
+ module. It prepares all arguments required by sphinx-build.
+ """
+
+ builder = TARGETS[target]["builder"]
+ out_dir = TARGETS[target].get("out_dir", "")
+
+ # Cleandocs doesn't require sphinx-build
+ if target == "cleandocs":
+ self.cleandocs(builder)
+ return
+
+ # Other targets require sphinx-build
+ sphinxbuild = shutil.which(self.sphinxbuild, path=self.env["PATH"])
+ if not sphinxbuild:
+ sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n")
+
+ self.prepare_media(builder)
+
+ if builder == "latex":
+ if not self.pdflatex_cmd and not self.latexmk_cmd:
+ sys.exit("Error: pdflatex or latexmk required for PDF generation")
+
+ docs_dir = os.path.abspath(os.path.join(self.srctree, "Documentation"))
+
+ # Prepare base arguments for Sphinx build
+ kerneldoc = self.kerneldoc
+ if kerneldoc.startswith(self.srctree):
+ kerneldoc = os.path.relpath(kerneldoc, self.srctree)
+
+ # Prepare common Sphinx options
+ args = [
+ "-b", builder,
+ "-c", docs_dir,
+ ]
+
+ if builder == "latex":
+ if not paper:
+ paper = PAPER[1]
+
+ args.extend(["-D", f"latex_elements.papersize={paper}paper"])
+
+ if not self.verbose:
+ args.append("-q")
+
+ if self.config_rust:
+ args.extend(["-t", "rustdoc"])
+
+ if conf:
+ self.env["SPHINX_CONF"] = self.get_path(conf, abs_path=True)
+
+ if not sphinxdirs:
+ sphinxdirs = os.environ.get("SPHINXDIRS", ".")
+
+ # sphinxdirs can be a list or a whitespace-separated string
+ sphinxdirs_list = []
+ for sphinxdir in sphinxdirs:
+ if isinstance(sphinxdir, list):
+ sphinxdirs_list += sphinxdir
+ else:
+ for name in sphinxdir.split(" "):
+ sphinxdirs_list.append(name)
+
+ # Build each directory
+ output_dirs = []
+ for sphinxdir in sphinxdirs_list:
+ src_dir = os.path.join(docs_dir, sphinxdir)
+ doctree_dir = os.path.join(self.builddir, ".doctrees")
+ output_dir = os.path.join(self.builddir, sphinxdir, out_dir)
+
+ # Make directory names canonical
+ src_dir = os.path.normpath(src_dir)
+ doctree_dir = os.path.normpath(doctree_dir)
+ output_dir = os.path.normpath(output_dir)
+
+ os.makedirs(doctree_dir, exist_ok=True)
+ os.makedirs(output_dir, exist_ok=True)
+
+ output_dirs.append(output_dir)
+
+ build_args = args + [
+ "-d", doctree_dir,
+ "-D", f"kerneldoc_bin={kerneldoc}",
+ "-D", f"version={self.kernelversion}",
+ "-D", f"release={self.kernelrelease}",
+ "-D", f"kerneldoc_srctree={self.srctree}",
+ src_dir,
+ output_dir,
+ ]
+
+ # Execute sphinx-build
+ try:
+ self.run_sphinx(sphinxbuild, build_args, env=self.env)
+ except Exception as e:
+ sys.exit(f"Build failed: {e}")
+
+ # Ensure that html/epub will have needed static files
+ if target in ["htmldocs", "epubdocs"]:
+ self.handle_html(css, output_dir)
+
+ # PDF and Info require a second build step
+ if target == "pdfdocs":
+ self.handle_pdf(output_dirs)
+ elif target == "infodocs":
+ self.handle_info(output_dirs)
+
+ @staticmethod
+ def get_python_version(cmd):
+ """
+ Get python version from a Python binary. As we need to detect if
+ are out there newer python binaries, we can't rely on sys.release here.
+ """
+
+ result = subprocess.run([cmd, "--version"], check=True,
+ stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ universal_newlines=True)
+ version = result.stdout.strip()
+
+ match = re.search(r"(\d+\.\d+\.\d+)", version)
+ if match:
+ return parse_version(match.group(1))
+
+ print(f"Can't parse version {version}")
+ return (0, 0, 0)
+
+ @staticmethod
+ def find_python():
+ """
+ Detect if are out there any python 3.xy version newer than the
+ current one.
+
+ Note: this routine is limited to up to 2 digits for python3. We
+ may need to update it one day, hopefully on a distant future.
+ """
+ patterns = [
+ "python3.[0-9]",
+ "python3.[0-9][0-9]",
+ ]
+
+ # Seek for a python binary newer than MIN_PYTHON_VERSION
+ for path in os.getenv("PATH", "").split(":"):
+ for pattern in patterns:
+ for cmd in glob(os.path.join(path, pattern)):
+ if os.path.isfile(cmd) and os.access(cmd, os.X_OK):
+ version = SphinxBuilder.get_python_version(cmd)
+ if version >= MIN_PYTHON_VERSION:
+ return cmd
+
+ return None
+
+ @staticmethod
+ def check_python():
+ """
+ Check if the current python binary satisfies our minimal requirement
+ for Sphinx build. If not, re-run with a newer version if found.
+ """
+ cur_ver = sys.version_info[:3]
+ if cur_ver >= MIN_PYTHON_VERSION:
+ return
+
+ python_ver = ver_str(cur_ver)
+
+ new_python_cmd = SphinxBuilder.find_python()
+ if not new_python_cmd:
+ sys.exit(f"Python version {python_ver} is not supported anymore.")
+
+ # Restart script using the newer version
+ script_path = os.path.abspath(sys.argv[0])
+ args = [new_python_cmd, script_path] + sys.argv[1:]
+
+ print(f"Python {python_ver} not supported. Changing to {new_python_cmd}")
+
+ try:
+ os.execv(new_python_cmd, args)
+ except OSError as e:
+ sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
+
+def main():
+ """
+ Main function. The only mandatory argument is the target. If not
+ specified, the other arguments will use default values if not
+ specified at os.environ.
+ """
+ parser = argparse.ArgumentParser(description="Kernel documentation builder")
+
+ parser.add_argument("target", choices=list(TARGETS.keys()),
+ help="Documentation target to build")
+ parser.add_argument("--sphinxdirs", nargs="+",
+ help="Specific directories to build")
+ parser.add_argument("--conf", default="conf.py",
+ help="Sphinx configuration file")
+
+ parser.add_argument("--theme", help="Sphinx theme to use")
+
+ parser.add_argument("--css", help="Custom CSS file for HTML/EPUB")
+
+ parser.add_argument("--paper", choices=PAPER, default=PAPER[0],
+ help="Paper size for LaTeX/PDF output")
+
+ parser.add_argument("-v", "--verbose", action='store_true',
+ help="place build in verbose mode")
+
+ parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
+ default=None,
+ help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
+
+ args = parser.parse_args()
+
+ if not args.verbose:
+ args.verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
+
+ SphinxBuilder.check_python()
+
+ builder = SphinxBuilder(venv=args.venv, verbose=args.verbose)
+
+ builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
+ theme=args.theme, css=args.css, paper=args.paper)
+
+if __name__ == "__main__":
+ main()
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-15 11:50 ` [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
@ 2025-08-16 1:16 ` Akira Yokosawa
2025-08-16 11:06 ` Mauro Carvalho Chehab
2025-08-21 19:36 ` Jonathan Corbet
2025-08-21 20:11 ` Jonathan Corbet
2 siblings, 1 reply; 21+ messages in thread
From: Akira Yokosawa @ 2025-08-16 1:16 UTC (permalink / raw)
To: mchehab+huawei
Cc: alex.gaynor, aliceryhl, bjorn3_gh, boqun.feng, corbet, gary,
linux-doc, linux-kernel, rust-for-linux, tmgross, Akira Yokosawa
Hi Mauro,
On Fri, 15 Aug 2025 13:50:32 +0200, Mauro Carvalho Chehab wrote:
> There are too much magic inside docs Makefile to properly run
> sphinx-build. Create an ancillary script that contains all
> kernel-related sphinx-build call logic currently at Makefile.
>
> Such script is designed to work both as an standalone command
> and as part of a Makefile. As such, it properly handles POSIX
> jobserver used by GNU make.
>
> It should be noticed that, when running the script alone,
> it will only take care of sphinx-build and cleandocs target.
> As such:
>
> - it won't run "make rustdoc";
> - no extra checks.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> .pylintrc | 2 +-
> scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++
> 2 files changed, 628 insertions(+), 1 deletion(-)
> create mode 100755 scripts/sphinx-build-wrapper
>
[...]
> diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper
> new file mode 100755
> index 000000000000..5c728956b53c
> --- /dev/null
> +++ b/scripts/sphinx-build-wrapper
> @@ -0,0 +1,627 @@
[...]
> + def handle_pdf(self, output_dirs):
> + """
> + Extra steps for PDF output.
> +
> + As PDF is handled via a LaTeX output, after building the .tex file,
> + a new build is needed to create the PDF output from the latex
> + directory.
> + """
> + builds = {}
> + max_len = 0
> +
> + for from_dir in output_dirs:
> + pdf_dir = os.path.join(from_dir, "../pdf")
> + os.makedirs(pdf_dir, exist_ok=True)
> +
> + if self.latexmk_cmd:
> + latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
> + else:
> + latex_cmd = [self.pdflatex]
> +
> + latex_cmd.extend(shlex.split(self.latexopts))
> +
> + tex_suffix = ".tex"
> +
> + # Process each .tex file
> + has_tex = False
> + build_failed = False
> + with os.scandir(from_dir) as it:
> + for entry in it:
> + if not entry.name.endswith(tex_suffix):
> + continue
> +
> + name = entry.name[:-len(tex_suffix)]
> + has_tex = True
> +
> + try:
> + subprocess.run(latex_cmd + [entry.path],
> + cwd=from_dir, check=True)
So, runs of latexmk (or xelatex) would be serialized, wouldn't they?
That would be a *huge* performance regression when I say:
"make -j10 -O pdfdocs"
Current Makefile delegates .tex --> .pdf part of pdfdocs to sub make
of .../output/Makefile, which is generated on-the-fly by Sphinx's
latex builder. That "-j10 -O" flag is passed to the sub make.
Another issue is that you are not deny-listing variable Noto CJK
fonts for latexmk/xelatex. So this version of wrapper won't be able
to build translations.pdf if you have such variable fonts installed.
That failuer is not caught by your ad-hoc logic below.
> + except subprocess.CalledProcessError:
> + # LaTeX PDF error code is almost useless: it returns
> + # error codes even when build succeeds but has warnings.
> + pass
> +
> + # Instead of checking errors, let's do the next best thing:
> + # check if the PDF file was actually created.
I've seen cases where a corrupt .pdf file is left after premature crashes
of xdvipdfmx. So, checking .pdf is not good enough for determining
success/failure.
One way to see if a .pdf file is properly formatted would be to run
"pdffonts" against the resulting .pdf.
For example, if I run "pdffonts translations.pdf" against the corrupted
one, I get:
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't read xref table
, with the exit code of 1.
Against a successfully built translations.pdf, I get something like:
name type encoding emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
JPRCQB+DejaVuSans-Bold CID TrueType Identity-H yes yes yes 4 0
QFNXFP+DejaVuSerif-Bold CID TrueType Identity-H yes yes yes 13 0
NMFBZR+NotoSerifCJKjp-Bold-Identity-H CID Type 0C Identity-H yes yes yes 15 0
WYMCYC+NotoSansCJKjp-Black-Identity-H CID Type 0C Identity-H yes yes yes 32 0
[...]
So I have to say this version of your wrapper does not look quite
ready to replace what you call "too much magic inside docs Makefile".
Regards,
Akira
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-16 1:16 ` Akira Yokosawa
@ 2025-08-16 11:06 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-16 11:06 UTC (permalink / raw)
To: Akira Yokosawa
Cc: alex.gaynor, aliceryhl, bjorn3_gh, boqun.feng, corbet, gary,
linux-doc, linux-kernel, rust-for-linux, tmgross
Em Sat, 16 Aug 2025 10:16:01 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:
> Hi Mauro,
>
> On Fri, 15 Aug 2025 13:50:32 +0200, Mauro Carvalho Chehab wrote:
> > There are too much magic inside docs Makefile to properly run
> > sphinx-build. Create an ancillary script that contains all
> > kernel-related sphinx-build call logic currently at Makefile.
> >
> > Such script is designed to work both as an standalone command
> > and as part of a Makefile. As such, it properly handles POSIX
> > jobserver used by GNU make.
> >
> > It should be noticed that, when running the script alone,
> > it will only take care of sphinx-build and cleandocs target.
> > As such:
> >
> > - it won't run "make rustdoc";
> > - no extra checks.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > .pylintrc | 2 +-
> > scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++
> > 2 files changed, 628 insertions(+), 1 deletion(-)
> > create mode 100755 scripts/sphinx-build-wrapper
> >
>
> [...]
>
> > diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper
> > new file mode 100755
> > index 000000000000..5c728956b53c
> > --- /dev/null
> > +++ b/scripts/sphinx-build-wrapper
> > @@ -0,0 +1,627 @@
>
> [...]
>
> > + def handle_pdf(self, output_dirs):
> > + """
> > + Extra steps for PDF output.
> > +
> > + As PDF is handled via a LaTeX output, after building the .tex file,
> > + a new build is needed to create the PDF output from the latex
> > + directory.
> > + """
> > + builds = {}
> > + max_len = 0
> > +
> > + for from_dir in output_dirs:
> > + pdf_dir = os.path.join(from_dir, "../pdf")
> > + os.makedirs(pdf_dir, exist_ok=True)
> > +
> > + if self.latexmk_cmd:
> > + latex_cmd = [self.latexmk_cmd, f"-{self.pdflatex}"]
> > + else:
> > + latex_cmd = [self.pdflatex]
> > +
> > + latex_cmd.extend(shlex.split(self.latexopts))
> > +
> > + tex_suffix = ".tex"
> > +
> > + # Process each .tex file
> > + has_tex = False
> > + build_failed = False
> > + with os.scandir(from_dir) as it:
> > + for entry in it:
> > + if not entry.name.endswith(tex_suffix):
> > + continue
> > +
> > + name = entry.name[:-len(tex_suffix)]
> > + has_tex = True
> > +
> > + try:
> > + subprocess.run(latex_cmd + [entry.path],
> > + cwd=from_dir, check=True)
>
> So, runs of latexmk (or xelatex) would be serialized, wouldn't they?
I guess it is already serialized today, but haven't check. IMO,
not serializing it is not a good idea, due to the very noisy build
that LaTeX does.
> That would be a *huge* performance regression when I say:
>
> "make -j10 -O pdfdocs"
>
> Current Makefile delegates .tex --> .pdf part of pdfdocs to sub make
> of .../output/Makefile, which is generated on-the-fly by Sphinx's
> latex builder. That "-j10 -O" flag is passed to the sub make.
Well, the script could pick it from jobserver with something
like (untested):
from concurrent import futures
# When using make, this won't be used, as the number of jobs comes
# from POSIX jobserver. So, this covers the case where build comes
# from command line. On such case, serialize by default, except if
# the user explicitly sets the number of jobs.
n_jobs = 1
if self.n_jobs:
try:
n_jobs = int(self.n_jobs)
except ValueError:
pass
with JobserverExec() as jobserver:
if jobserver.claim:
n_jobs = str(jobserver.claim)
fut = []
with futures.ThreadPoolExecutor(max_workers=self.MAX_WORKERS) as ex:
fut.append(ex.submit(self.handle_pdf(output_dirs))
for f in futures.as_completed(fut):
# some logic to pick job results and wait for them to comlete
This way, when running from command line without "-j" it would serialize
(with is good for debugging). Otherwise, it will pick the maximum number
of available jobs or the -j argument and create one process for each.
> Another issue is that you are not deny-listing variable Noto CJK
> fonts for latexmk/xelatex. So this version of wrapper won't be able
> to build translations.pdf if you have such variable fonts installed.
It did build it on several distributions:
almalinux_report.log: translations : PASSED: pdf/translations.pdf
amazonlinux_report.log: translations : PASSED: pdf/translations.pdf
centos_report.log: translations : PASSED: pdf/translations.pdf
fedora_report.log: translations : PASSED: pdf/translations.pdf
kali_report.log: translations : PASSED: pdf/translations.pdf
mageia_report.log: translations : PASSED: pdf/translations.pdf
opensuse-leap_report.log: translations : PASSED: pdf/translations.pdf
opensuse_report.log: translations : PASSED: pdf/translations.pdf
rockylinux8_report.log: translations : PASSED: pdf/translations.pdf
ubuntu_report.log: translations : PASSED: pdf/translations.pdf
(to test, you need to have both this series and the pdf fixes one
altogether - as I found some bugs related to it that were addressed
there)
> That failuer is not caught by your ad-hoc logic below.
>
> > + except subprocess.CalledProcessError:
> > + # LaTeX PDF error code is almost useless: it returns
> > + # error codes even when build succeeds but has warnings.
> > + pass
> > +
> > + # Instead of checking errors, let's do the next best thing:
> > + # check if the PDF file was actually created.
It is intentional. Even when everything passes, because of the
thousands of warnings, this always return an error code.
What the logic does, instead, is to ignore the bogus return code from
PDF builds, looking, instead if file is present, reporting, at the end:
Summary
=======
dev-tools : pdf/dev-tools.pdf
tools : pdf/tools.pdf
filesystems : pdf/filesystems.pdf
w1 : pdf/w1.pdf
maintainer : pdf/maintainer.pdf
process : pdf/process.pdf
isdn : pdf/isdn.pdf
fault-injection: pdf/fault-injection.pdf
iio : pdf/iio.pdf
scheduler : pdf/scheduler.pdf
staging : pdf/staging.pdf
fpga : pdf/fpga.pdf
power : pdf/power.pdf
leds : pdf/leds.pdf
edac : pdf/edac.pdf
PCI : pdf/PCI.pdf
firmware-guide : pdf/firmware-guide.pdf
cpu-freq : pdf/cpu-freq.pdf
mhi : pdf/mhi.pdf
wmi : pdf/wmi.pdf
timers : pdf/timers.pdf
accel : pdf/accel.pdf
hid : pdf/hid.pdf
userspace-api : pdf/userspace-api.pdf
spi : pdf/spi.pdf
networking : pdf/networking.pdf
virt : pdf/virt.pdf
nvme : pdf/nvme.pdf
translations : pdf/translations.pdf
input : pdf/input.pdf
tee : pdf/tee.pdf
doc-guide : pdf/doc-guide.pdf
cdrom : pdf/cdrom.pdf
gpu : pdf/gpu.pdf
i2c : pdf/i2c.pdf
RCU : pdf/RCU.pdf
watchdog : pdf/watchdog.pdf
usb : pdf/usb.pdf
rust : pdf/rust.pdf
crypto : pdf/crypto.pdf
kbuild : pdf/kbuild.pdf
livepatch : pdf/livepatch.pdf
mm : pdf/mm.pdf
locking : pdf/locking.pdf
infiniband : pdf/infiniband.pdf
driver-api : pdf/driver-api.pdf
bpf : pdf/bpf.pdf
devicetree : pdf/devicetree.pdf
block : pdf/block.pdf
target : pdf/target.pdf
arch : pdf/arch.pdf
pcmcia : pdf/pcmcia.pdf
scsi : pdf/scsi.pdf
netlabel : pdf/netlabel.pdf
sound : pdf/sound.pdf
security : pdf/security.pdf
accounting : pdf/accounting.pdf
admin-guide : pdf/admin-guide.pdf
core-api : pdf/core-api.pdf
fb : pdf/fb.pdf
peci : pdf/peci.pdf
trace : pdf/trace.pdf
misc-devices : pdf/misc-devices.pdf
kernel-hacking : pdf/kernel-hacking.pdf
hwmon : pdf/hwmon.pdf
and returning 0 if all of the above were built. At the above,
all PDF files were built.
If one of the files is not built, it reports, instead:
...
accel : pdf/accel.pdf
hid : pdf/hid.pdf
userspace-api : FAILED
spi : pdf/spi.pdf
networking : pdf/networking.pdf
virt : pdf/virt.pdf
nvme : pdf/nvme.pdf
translations : FAILED
input : pdf/inp
...
In this specific example (pick from Mint build), userspace-api and
translations failed, among others:
$ grep -Ei "^\w.*: FAILED" mint_report.log
userspace-api : FAILED
translations : FAILED
doc-guide : FAILED
gpu : FAILED
i2c : FAILED
RCU : FAILED
arch : FAILED
core-api : FAILED
Clearly, the failure there were not specific to translations,
and it is distro-specific.
One of the possible causes could be distro specific LaTeX config limits
for things like secnumdepth, tocdepth, ... and memory limits.
For instance, Mint has this main memory limit:
/usr/share/texlive/texmf-dist/web2c/texmf.cnf:main_memory = 5000000 % words of inimemory available; also applies to inimf&mp
while Fedora uses a higher limit:
/etc/texlive/web2c/texmf.cnf:main_memory = 6000000 % words of inimemory available; also applies to inimf&mp
In the past, I had to fix several of those to generate PDFs on
Debian 11 and older Fedora. The fix is distro-specific.
-
If you look at the results I placed at the pdf series, on those distros,
all PDF files including translations were built:
PASSED - AlmaLinux release 9.6 (Sage Margay) (7 tests)
PASSED - Amazon Linux release 2023 (Amazon Linux) (7 tests)
PASSED - CentOS Stream release 9 (7 tests)
PASSED - Fedora release 42 (Adams) (7 tests)
PASSED - Kali GNU/Linux 2025.2 (7 tests)
PASSED - Mageia 9 (7 tests)
PASSED - openSUSE Leap 15.6 (7 tests)
PASSED - openSUSE Tumbleweed (7 tests)
PASSED - Ubuntu 25.04 (7 tests)
> I've seen cases where a corrupt .pdf file is left after premature crashes
> of xdvipdfmx. So, checking .pdf is not good enough for determining
> success/failure.
>
> One way to see if a .pdf file is properly formatted would be to run
> "pdffonts" against the resulting .pdf.
>
> For example, if I run "pdffonts translations.pdf" against the corrupted
> one, I get:
>
> Syntax Error: Couldn't find trailer dictionary
> Syntax Error: Couldn't find trailer dictionary
> Syntax Error: Couldn't read xref table
>
> , with the exit code of 1.
> Against a successfully built translations.pdf, I get something like:
>
> name type encoding emb sub uni object ID
> ------------------------------------ ----------------- ---------------- --- --- --- ---------
> JPRCQB+DejaVuSans-Bold CID TrueType Identity-H yes yes yes 4 0
> QFNXFP+DejaVuSerif-Bold CID TrueType Identity-H yes yes yes 13 0
> NMFBZR+NotoSerifCJKjp-Bold-Identity-H CID Type 0C Identity-H yes yes yes 15 0
> WYMCYC+NotoSansCJKjp-Black-Identity-H CID Type 0C Identity-H yes yes yes 32 0
> [...]
>
Running it at the Fedora container shows that CJK fonts are there:
$ pdffonts Documentation/output/pdf/translations.pdf
name type encoding emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
JOMCYL+DejaVuSans-Bold CID TrueType Identity-H yes yes yes 4 0
HKSYWD+DejaVuSerif-Bold CID TrueType Identity-H yes yes yes 13 0
TMISTH+NotoSansCJKjp-Black-Identity-H CID Type 0C Identity-H yes yes yes 15 0
FCMOXF+NotoSansCJKjp-Black-Identity-H CID Type 0C Identity-H yes yes yes 32 0
IPEVCV+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 34 0
UHCJPQ+DejaVuSerif CID TrueType Identity-H yes yes yes 36 0
ZPKIZY+DejaVuSerif-Italic CID TrueType Identity-H yes yes yes 41 0
QIGBJX+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 44 0
FHPART+CMMI6 Type 1C Builtin yes yes no 244 0
DHNQVK+CMSY6 Type 1C Builtin yes yes yes 245 0
OFGXYL+DejaVuSerif-BoldItalic CID TrueType Identity-H yes yes yes 415 0
SKLJTY+NotoSansCJKjp-Bold-Identity-H CID Type 0C Identity-H yes yes yes 503 0
JXTNEV+MSAM10 Type 1C Builtin yes yes yes 618 0
KEIGKE+CMSY10 Type 1C Builtin yes yes yes 637 0
GOHSWX+DejaVuSans CID TrueType Identity-H yes yes yes 941 0
IVVPKU+CMMI10 Type 1C Builtin yes yes yes 1592 0
XAQAVF+CMR10 Type 1C Builtin yes yes yes 1593 0
TAUDRW+CMR8 Type 1C Builtin yes yes yes 1594 0
AMQGFF+CMMI8 Type 1C Builtin yes yes yes 1595 0
ETSVZG+CMSY8 Type 1C Builtin yes yes yes 1596 0
NREKPL+NimbusRoman-Regular Type 1C WinAnsi yes yes yes 2508 0
ZYNGMU+NotoSansCJKjp-Regular CID Type 0C Identity-H yes yes yes 2519 0
EITFCN+NotoSansCJKjp-Black-Identity-H CID Type 0C Identity-H yes yes yes 3782 0
ZDUUCF+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 3787 0
SAYAHH+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 3848 0
QFVHQT+NotoSansCJKjp-Bold-Identity-H CID Type 0C Identity-H yes yes yes 4172 0
OGQDEG+DejaVuSansMono CID TrueType Identity-H yes yes yes 5311 0
QFJUVY+DejaVuSans-BoldOblique CID TrueType Identity-H yes yes yes 5502 0
JYGYZI+DejaVuSansMono-Bold CID TrueType Identity-H yes yes yes 5607 0
UBBPKA+DejaVuSansMono-Oblique CID TrueType Identity-H yes yes yes 5733 0
LRVSWG+NimbusRoman-Regular Type 1C WinAnsi yes yes yes 6299 0
KCVHZZ+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 6583 0
NKLOXT+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 6602 0
URSEFI+NotoSansCJKjp-Black-Identity-H CID Type 0C Identity-H yes yes yes 6814 0
MDHJPT+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 6819 0
UVENXR+NotoSansCJKjp-Regular-Identity-H CID Type 0C Identity-H yes yes yes 6841 0
On Leap, It reports:
opensuse-leap-test:~ # pdffonts Documentation/output/pdf/translations.pdf
name type encoding emb sub uni object ID
------------------------------------ ----------------- ---------------- --- --- --- ---------
JXGSQR+DejaVuSans-Bold CID TrueType Identity-H yes yes yes 4 0
HQDZTS+DejaVuSerif-Bold CID TrueType Identity-H yes yes yes 13 0
RHJSVS+DejaVuSerif CID TrueType Identity-H yes yes yes 26 0
YMIDKW+DejaVuSansMono CID TrueType Identity-H yes yes yes 28 0
AQCTYG+DejaVuSerif-Italic CID TrueType Identity-H yes yes yes 37 0
DYBJGY+CMMI6 Type 1C Builtin yes yes no 166 0
PVSXRL+CMSY6 Type 1C Builtin yes yes yes 167 0
OKDRKF+DejaVuSans-BoldOblique CID TrueType Identity-H yes yes yes 294 0
UATVDQ+DejaVuSerif-BoldItalic CID TrueType Identity-H yes yes yes 342 0
HMMWHY+DejaVuSansMono-Bold CID TrueType Identity-H yes yes yes 401 0
ZNFJHG+DejaVuSans CID TrueType Identity-H yes yes yes 408 0
YVEKLE+DejaVuSansMono-Oblique CID TrueType Identity-H yes yes yes 529 0
RKPRBN+MSAM10 Type 1C Builtin yes yes yes 625 0
AKVBFL+CMSY10 Type 1C Builtin yes yes yes 640 0
QFXNIX+TeXGyreTermes-Regular Type 1C WinAnsi yes yes yes 1103 0
PZNVGF+TeXGyreTermes-Regular Type 1C WinAnsi yes yes yes 1111 0
No CJK fonts there, which is already expected, as fonts were installed per sphinx-pre-install,
which explicitly says it currently doesn't add CJK fonts on OpenSUSE:
# FIXME: add support for installing CJK fonts
#
# I tried hard, but was unable to find a way to install
# "Noto Sans CJK SC" on openSUSE
For both cases, pdffonts didn't report any syntax error warnings.
So, for me it is working fine on both cases: when CJK is available and
when it is missing.
-
Now, with regards of automating a check with pdffonts, current
build process doesn't do it; it follows the typical assumption
that, when a make target produces a file, such file is not
corrupted.
Now, I don't mind having a consistency check after the build to
verify if the pdf file is corrupted, but such kind of things
deserve its own specific series.
As you know a lot more about that and CJK specific dependencies,
if you think this is important, feel free to could craft such test
and add it later at the wrapper. On such series, it would also
be worth to suggest installing poppler-utils/poppler-tools at
scripts/sphinx-pre-install.
The logic at sphinx-build-wrapper would then check if pdftools
is installed, before running the consistency checks that need it.
> So I have to say this version of your wrapper does not look quite
> ready to replace what you call "too much magic inside docs Makefile".
From what I got, the only missing piece is the parallel build.
I'll craft an extra patch for it.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-15 11:50 ` [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
2025-08-16 1:16 ` Akira Yokosawa
@ 2025-08-21 19:36 ` Jonathan Corbet
2025-08-21 19:43 ` Mauro Carvalho Chehab
2025-08-21 20:11 ` Jonathan Corbet
2 siblings, 1 reply; 21+ messages in thread
From: Jonathan Corbet @ 2025-08-21 19:36 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron,
Mauro Carvalho Chehab, Alex Gaynor, Alice Ryhl, Andreas Hindborg,
Benno Lossin, Boqun Feng, Danilo Krummrich, Gary Guo,
Miguel Ojeda, Trevor Gross, linux-kernel, rust-for-linux
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> There are too much magic inside docs Makefile to properly run
> sphinx-build. Create an ancillary script that contains all
> kernel-related sphinx-build call logic currently at Makefile.
So I am just now looking at the script and seeking to understand it, but
one thing has jumped at me that I wanted to toss out there...
> +# Minimal supported Python version needed by Sphinx and its extensions
> +MIN_PYTHON_VERSION = parse_version("3.7")
> +
> +# Default value for --venv parameter
> +VENV_DEFAULT = "sphinx_latest"
> +
> +# List of make targets and its corresponding builder and output directory
> +TARGETS = {
We don't at this point have a formal coding standard for Python code,
but I do think that we should, to the extent possible, stick to the
rules that have been established for C code. One thing I would really
like to see is in the comment style; our rules want:
/*
* ...C comments spread out with the markers on separate lines
* like this...
*/
so can we do something similar for Python?
#
# Markers above and below
#
I will confess that this matches my personal subject preference, but it
also brings us closer to what our C code looks like.
(I don't know that I would push to redo everything to match that style,
but instead to move that way going forward).
Thanks,
jon
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-21 19:36 ` Jonathan Corbet
@ 2025-08-21 19:43 ` Mauro Carvalho Chehab
2025-08-21 20:18 ` Jonathan Corbet
0 siblings, 1 reply; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-21 19:43 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Linux Doc Mailing List, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross, linux-kernel,
rust-for-linux
Em Thu, 21 Aug 2025 13:36:24 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
> > There are too much magic inside docs Makefile to properly run
> > sphinx-build. Create an ancillary script that contains all
> > kernel-related sphinx-build call logic currently at Makefile.
>
> So I am just now looking at the script and seeking to understand it, but
> one thing has jumped at me that I wanted to toss out there...
>
> > +# Minimal supported Python version needed by Sphinx and its extensions
> > +MIN_PYTHON_VERSION = parse_version("3.7")
> > +
> > +# Default value for --venv parameter
> > +VENV_DEFAULT = "sphinx_latest"
> > +
> > +# List of make targets and its corresponding builder and output directory
> > +TARGETS = {
>
> We don't at this point have a formal coding standard for Python code,
> but I do think that we should, to the extent possible, stick to the
> rules that have been established for C code. One thing I would really
> like to see is in the comment style; our rules want:
>
> /*
> * ...C comments spread out with the markers on separate lines
> * like this...
> */
>
> so can we do something similar for Python?
>
> #
> # Markers above and below
> #
>
> I will confess that this matches my personal subject preference, but it
> also brings us closer to what our C code looks like.
Fine for me. Can I do such changes on a patch at the end of the series
to prevent rebase conflicts?
> (I don't know that I would push to redo everything to match that style,
> but instead to move that way going forward).
>
> Thanks,
>
> jon
Thanks,
Mauro
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-21 19:43 ` Mauro Carvalho Chehab
@ 2025-08-21 20:18 ` Jonathan Corbet
0 siblings, 0 replies; 21+ messages in thread
From: Jonathan Corbet @ 2025-08-21 20:18 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Linux Doc Mailing List, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross, linux-kernel,
rust-for-linux
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> Em Thu, 21 Aug 2025 13:36:24 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>>
>> > There are too much magic inside docs Makefile to properly run
>> > sphinx-build. Create an ancillary script that contains all
>> > kernel-related sphinx-build call logic currently at Makefile.
>>
>> So I am just now looking at the script and seeking to understand it, but
>> one thing has jumped at me that I wanted to toss out there...
>>
>> > +# Minimal supported Python version needed by Sphinx and its extensions
>> > +MIN_PYTHON_VERSION = parse_version("3.7")
>> > +
>> > +# Default value for --venv parameter
>> > +VENV_DEFAULT = "sphinx_latest"
>> > +
>> > +# List of make targets and its corresponding builder and output directory
>> > +TARGETS = {
>>
>> We don't at this point have a formal coding standard for Python code,
>> but I do think that we should, to the extent possible, stick to the
>> rules that have been established for C code. One thing I would really
>> like to see is in the comment style; our rules want:
>>
>> /*
>> * ...C comments spread out with the markers on separate lines
>> * like this...
>> */
>>
>> so can we do something similar for Python?
>>
>> #
>> # Markers above and below
>> #
>>
>> I will confess that this matches my personal subject preference, but it
>> also brings us closer to what our C code looks like.
>
> Fine for me. Can I do such changes on a patch at the end of the series
> to prevent rebase conflicts?
Yes, of course - and I don't think we have to fix everything right away
either. We already have plenty of inconsistent stuff, we can deal with
it in the usual manner, cleaning it up when we're in the neighborhood
anyway.
Thanks,
jon
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-15 11:50 ` [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
2025-08-16 1:16 ` Akira Yokosawa
2025-08-21 19:36 ` Jonathan Corbet
@ 2025-08-21 20:11 ` Jonathan Corbet
2025-08-22 2:06 ` Mauro Carvalho Chehab
2 siblings, 1 reply; 21+ messages in thread
From: Jonathan Corbet @ 2025-08-21 20:11 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron,
Mauro Carvalho Chehab, Alex Gaynor, Alice Ryhl, Andreas Hindborg,
Benno Lossin, Boqun Feng, Danilo Krummrich, Gary Guo,
Miguel Ojeda, Trevor Gross, linux-kernel, rust-for-linux
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> There are too much magic inside docs Makefile to properly run
> sphinx-build. Create an ancillary script that contains all
> kernel-related sphinx-build call logic currently at Makefile.
>
> Such script is designed to work both as an standalone command
> and as part of a Makefile. As such, it properly handles POSIX
> jobserver used by GNU make.
>
> It should be noticed that, when running the script alone,
> it will only take care of sphinx-build and cleandocs target.
> As such:
>
> - it won't run "make rustdoc";
> - no extra checks.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> .pylintrc | 2 +-
> scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++
> 2 files changed, 628 insertions(+), 1 deletion(-)
> create mode 100755 scripts/sphinx-build-wrapper
As a whole I like the idea of this - I would rather be reading code in
Python than in makefilese. But I have some overall notes...
I am a bit dismayed by the size of it; this is many times the amount of
code it allows us to remove from the makefile. Perhaps there's nothing
to be done for that, but ...
Is there value in the SphinxBuilder class? Just because you can create
classes in Python doesn't mean that you have to; I'm not sure why you
would create one here rather than just doing it all at the module level.
Is the "search for a newer Python" code really going to be useful for
anybody? It seems like a lot of work (and code) to try to quietly patch
things up for somebody who has some sort of a strange setup.
Please, no "except Exception:" (or the equivalent bare "except:").
That bit of locale tweaking shows up in enough places that it should
maybe go into a little helper module rather than being repeatedly
open-coded?
Thanks,
jon
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-21 20:11 ` Jonathan Corbet
@ 2025-08-22 2:06 ` Mauro Carvalho Chehab
2025-08-22 13:55 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-22 2:06 UTC (permalink / raw)
To: Jonathan Corbet
Cc: Linux Doc Mailing List, Björn Roy Baron, Alex Gaynor,
Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross, linux-kernel,
rust-for-linux
Em Thu, 21 Aug 2025 14:11:06 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
> > There are too much magic inside docs Makefile to properly run
> > sphinx-build. Create an ancillary script that contains all
> > kernel-related sphinx-build call logic currently at Makefile.
> >
> > Such script is designed to work both as an standalone command
> > and as part of a Makefile. As such, it properly handles POSIX
> > jobserver used by GNU make.
> >
> > It should be noticed that, when running the script alone,
> > it will only take care of sphinx-build and cleandocs target.
> > As such:
> >
> > - it won't run "make rustdoc";
> > - no extra checks.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > .pylintrc | 2 +-
> > scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++
> > 2 files changed, 628 insertions(+), 1 deletion(-)
> > create mode 100755 scripts/sphinx-build-wrapper
>
> As a whole I like the idea of this - I would rather be reading code in
> Python than in makefilese. But I have some overall notes...
>
> I am a bit dismayed by the size of it; this is many times the amount of
> code it allows us to remove from the makefile. Perhaps there's nothing
> to be done for that, but ...
True:
7 files changed, 922 insertions(+), 200 deletions(-)
Yet, a lot of them are blank lines, comments, docstrings and
argparse. Also, there are some improvements on it, like the PDF
build error handling logic (more below).
One of the things I wanted to do is precisely better describe
what it does. The Makefile has lots of magic and complex macros
that, even the ones that wrote it have headaches trying to understand
after some time not looking on it.
Also, part of it will be dropped at the next patch series when
we will get rid of userspace-api/media/Makefile:
$ git diff 182fa089db0b..parse_headers scripts/sphinx-build-wrapper|diffstat -p1
scripts/sphinx-build-wrapper | 48 ---------------------
1 file changed, 48 deletions(-)
I guess there are some space to clean it up a little bit, for instance
getting rid of most of the env vars and passing them as command
line only. Yet, I opted to be a bit conservative at the cleanups at the
env vars to reduce the risk of breaking something.
> Is there value in the SphinxBuilder class? Just because you can create
> classes in Python doesn't mean that you have to; I'm not sure why you
> would create one here rather than just doing it all at the module level.
On Python, I prefer using classes, as:
1. I don't like passing lots of arguments to functions;
2. Python doesn't have struct. The closest concept of struct in Python
is a data class;
3. Classes help to avoid global vars. In this specific case, the size of
its data "struct" (e.g. class variables) is not small, as it has to
parse lots of environment vars, and several of those are used on
multiple places.
> Is the "search for a newer Python" code really going to be useful for
> anybody?
Well, I could have written it in shell script or Perl ;-)
Seriously, the rationale was not to search for a newer Python code, but
instead, to have a place were it is easier to understand what's
going on and adjust to our actual needs.
I actually started it because of the pdfdocs check: doing make
pdfdocs currently will return an error code when building docs
no matter what. Fixing it inside the Makefile would be doable, but
complex and would likely require an script anyway (or a script-like
code embedded on it).
Now, instead of adding yet another hack there, why not do it
right?
> It seems like a lot of work (and code) to try to quietly patch
> things up for somebody who has some sort of a strange setup.
My idea was not to support some strange setup, but to I had a few
goals. Mainly:
1. to have the simplest possible Makefile without loosing anything;
2. to be able to call the script directly, as it helps debugging;
3. to remove that ugly for sphinx-build call macro logic inside
Makefile, with is hard to understand and even harder to touch;
4. to fix a known bug with the current approach with regards
to PDF build: no matter if PDF build succeeded or not, it
always return an error code;
5. to have a summary of the PDF build. Even with latexmk, the
PDF build is a way too noisy, being hard to check what broke
and what succeeded.
6. to make easier to build PDFs in interactive mode (I added this
later at the development cycle).
-
For the future, if time permits, there are two possible improvements
a) I'd like to change the way SPHINXDIRS work. Right now, it is a very
dirty hack, that makes sphinx-build becalled several times (one for
each dir), and breaks cross-references.
I'd like to be able to allow building multiple dirs at the
same time, with a proper index between them.
b) By having this in python, we can do other cleanups like:
- instance convert sphinx-pre-install into a class, calling
it directly there;
- pick the logic inside conf.py that handles SPHINXDIRS and
latex documents.
In summary, the advantage is that it is a lot easier to fine tune
the script in the proper way than to add more hacks to docs Makefile.
> Please, no "except Exception:" (or the equivalent bare "except:").
Ok.
> That bit of locale tweaking shows up in enough places that it should
> maybe go into a little helper module rather than being repeatedly
> open-coded?
Do you mean this?
# The sphinx-build tool has a bug: internally, it tries to set
# locale with locale.setlocale(locale.LC_ALL, ''). This causes a
# crash if language is not set. Detect and fix it.
try:
locale.setlocale(locale.LC_ALL, '')
except Exception: # I'll replace it with locale.Error
self.env["LC_ALL"] = "C"
self.env["LANG"] = "C"
This was also added later, to fix a current Sphinx (or Python?) bug.
Try to run sphinx-build on a Debian-based or Gentoo distro without
setting any locale (which is the default after installing them):
# sphinx-build --help
Traceback (most recent call last):
File "/usr/bin/sphinx-build", line 8, in <module>
sys.exit(main())
~~~~^^
File "/usr/lib/python3/dist-packages/sphinx/cmd/build.py", line 546, in main
locale.setlocale(locale.LC_ALL, '')
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/locale.py", line 615, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting
# LC_ALL=C sphinx-build --help
usage: sphinx-build [OPTIONS] SOURCEDIR OUTPUTDIR [FILENAMES...]
...
Well, sphinx-build will crash even if called with "--help". Btw,
spinx-build is following Python recommendation of using '':
https://docs.python.org/3/library/locale.html#locale.setlocale
Ok, we could drop that, but on the other hand it is just 5 LOC.
(actually it can be 4 LOC, as self.env["LANG"] = "C" is not really needed
to avoid the crash).
One thing I'm in doubt is with regards to --venv command line argument.
It could be dropped, or it could be auto-detected. The code is also
small, and IMHO it could help for the ones using venv:
# If venv parameter is specified, run Sphinx from venv
if venv:
bin_dir = os.path.join(venv, "bin")
if os.path.isfile(os.path.join(bin_dir, "activate")):
# "activate" virtual env
self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
self.env["VIRTUAL_ENV"] = venv
if "PYTHONHOME" in self.env:
del self.env["PYTHONHOME"]
print(f"Setting venv to {venv}")
else:
sys.exit(f"Venv {venv} not found.")
Btw, this is the kind of code that makes sense to be inside some
library.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build
2025-08-22 2:06 ` Mauro Carvalho Chehab
@ 2025-08-22 13:55 ` Mauro Carvalho Chehab
0 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-22 13:55 UTC (permalink / raw)
To: Mauro Carvalho Chehab
Cc: Jonathan Corbet, Linux Doc Mailing List, Björn Roy Baron,
Alex Gaynor, Alice Ryhl, Boqun Feng, Gary Guo, Trevor Gross,
linux-kernel, rust-for-linux
On Fri, Aug 22, 2025 at 04:06:45AM +0200, Mauro Carvalho Chehab wrote:
> Em Thu, 21 Aug 2025 14:11:06 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >
> > > There are too much magic inside docs Makefile to properly run
> > > sphinx-build. Create an ancillary script that contains all
> > > kernel-related sphinx-build call logic currently at Makefile.
> > >
> > > Such script is designed to work both as an standalone command
> > > and as part of a Makefile. As such, it properly handles POSIX
> > > jobserver used by GNU make.
> > >
> > > It should be noticed that, when running the script alone,
> > > it will only take care of sphinx-build and cleandocs target.
> > > As such:
> > >
> > > - it won't run "make rustdoc";
> > > - no extra checks.
> > >
> > > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > > ---
> > > .pylintrc | 2 +-
> > > scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++
> > > 2 files changed, 628 insertions(+), 1 deletion(-)
> > > create mode 100755 scripts/sphinx-build-wrapper
> >
> > As a whole I like the idea of this - I would rather be reading code in
> > Python than in makefilese. But I have some overall notes...
> >
> > I am a bit dismayed by the size of it; this is many times the amount of
> > code it allows us to remove from the makefile. Perhaps there's nothing
> > to be done for that, but ...
>
> True:
>
> 7 files changed, 922 insertions(+), 200 deletions(-)
>
> Yet, a lot of them are blank lines, comments, docstrings and
> argparse. Also, there are some improvements on it, like the PDF
> build error handling logic (more below).
>
> One of the things I wanted to do is precisely better describe
> what it does. The Makefile has lots of magic and complex macros
> that, even the ones that wrote it have headaches trying to understand
> after some time not looking on it.
>
> Also, part of it will be dropped at the next patch series when
> we will get rid of userspace-api/media/Makefile:
>
> $ git diff 182fa089db0b..parse_headers scripts/sphinx-build-wrapper|diffstat -p1
> scripts/sphinx-build-wrapper | 48 ---------------------
> 1 file changed, 48 deletions(-)
After sleeping on it, IMO the best would be if we change the order:
Let's place the series related to parse-headers.pl/kernel-images.py
first.
Then, on this series, I can suppress the code that has support for
the media build Makefile (reducing those 48 lines), making the
patches cleaner.
I'll also change this series to place the new wrapper under
tools/docs, splitting part of it on a library. The final code
will be bigger due to the library, but it will become more
modular and with a cleaner implementation.
>
> I guess there are some space to clean it up a little bit, for instance
> getting rid of most of the env vars and passing them as command
> line only. Yet, I opted to be a bit conservative at the cleanups at the
> env vars to reduce the risk of breaking something.
>
> > Is there value in the SphinxBuilder class? Just because you can create
> > classes in Python doesn't mean that you have to; I'm not sure why you
> > would create one here rather than just doing it all at the module level.
>
> On Python, I prefer using classes, as:
>
> 1. I don't like passing lots of arguments to functions;
> 2. Python doesn't have struct. The closest concept of struct in Python
> is a data class;
> 3. Classes help to avoid global vars. In this specific case, the size of
> its data "struct" (e.g. class variables) is not small, as it has to
> parse lots of environment vars, and several of those are used on
> multiple places.
>
> > Is the "search for a newer Python" code really going to be useful for
> > anybody?
>
> Well, I could have written it in shell script or Perl ;-)
>
> Seriously, the rationale was not to search for a newer Python code, but
> instead, to have a place were it is easier to understand what's
> going on and adjust to our actual needs.
>
> I actually started it because of the pdfdocs check: doing make
> pdfdocs currently will return an error code when building docs
> no matter what. Fixing it inside the Makefile would be doable, but
> complex and would likely require an script anyway (or a script-like
> code embedded on it).
>
> Now, instead of adding yet another hack there, why not do it
> right?
>
> > It seems like a lot of work (and code) to try to quietly patch
> > things up for somebody who has some sort of a strange setup.
>
> My idea was not to support some strange setup, but to I had a few
> goals. Mainly:
>
> 1. to have the simplest possible Makefile without loosing anything;
> 2. to be able to call the script directly, as it helps debugging;
> 3. to remove that ugly for sphinx-build call macro logic inside
> Makefile, with is hard to understand and even harder to touch;
> 4. to fix a known bug with the current approach with regards
> to PDF build: no matter if PDF build succeeded or not, it
> always return an error code;
> 5. to have a summary of the PDF build. Even with latexmk, the
> PDF build is a way too noisy, being hard to check what broke
> and what succeeded.
> 6. to make easier to build PDFs in interactive mode (I added this
> later at the development cycle).
>
> -
>
> For the future, if time permits, there are two possible improvements
>
> a) I'd like to change the way SPHINXDIRS work. Right now, it is a very
> dirty hack, that makes sphinx-build becalled several times (one for
> each dir), and breaks cross-references.
>
> I'd like to be able to allow building multiple dirs at the
> same time, with a proper index between them.
>
> b) By having this in python, we can do other cleanups like:
>
> - instance convert sphinx-pre-install into a class, calling
> it directly there;
> - pick the logic inside conf.py that handles SPHINXDIRS and
> latex documents.
>
> In summary, the advantage is that it is a lot easier to fine tune
> the script in the proper way than to add more hacks to docs Makefile.
>
> > Please, no "except Exception:" (or the equivalent bare "except:").
>
> Ok.
>
> > That bit of locale tweaking shows up in enough places that it should
> > maybe go into a little helper module rather than being repeatedly
> > open-coded?
>
> Do you mean this?
>
> # The sphinx-build tool has a bug: internally, it tries to set
> # locale with locale.setlocale(locale.LC_ALL, ''). This causes a
> # crash if language is not set. Detect and fix it.
> try:
> locale.setlocale(locale.LC_ALL, '')
> except Exception: # I'll replace it with locale.Error
> self.env["LC_ALL"] = "C"
> self.env["LANG"] = "C"
>
> This was also added later, to fix a current Sphinx (or Python?) bug.
>
> Try to run sphinx-build on a Debian-based or Gentoo distro without
> setting any locale (which is the default after installing them):
>
> # sphinx-build --help
> Traceback (most recent call last):
> File "/usr/bin/sphinx-build", line 8, in <module>
> sys.exit(main())
> ~~~~^^
> File "/usr/lib/python3/dist-packages/sphinx/cmd/build.py", line 546, in main
> locale.setlocale(locale.LC_ALL, '')
> ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
> File "/usr/lib/python3.13/locale.py", line 615, in setlocale
> return _setlocale(category, locale)
> locale.Error: unsupported locale setting
>
> # LC_ALL=C sphinx-build --help
> usage: sphinx-build [OPTIONS] SOURCEDIR OUTPUTDIR [FILENAMES...]
> ...
>
>
> Well, sphinx-build will crash even if called with "--help". Btw,
> spinx-build is following Python recommendation of using '':
> https://docs.python.org/3/library/locale.html#locale.setlocale
>
> Ok, we could drop that, but on the other hand it is just 5 LOC.
> (actually it can be 4 LOC, as self.env["LANG"] = "C" is not really needed
> to avoid the crash).
>
> One thing I'm in doubt is with regards to --venv command line argument.
> It could be dropped, or it could be auto-detected. The code is also
> small, and IMHO it could help for the ones using venv:
>
> # If venv parameter is specified, run Sphinx from venv
> if venv:
> bin_dir = os.path.join(venv, "bin")
> if os.path.isfile(os.path.join(bin_dir, "activate")):
> # "activate" virtual env
> self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
> self.env["VIRTUAL_ENV"] = venv
> if "PYTHONHOME" in self.env:
> del self.env["PYTHONHOME"]
> print(f"Setting venv to {venv}")
> else:
> sys.exit(f"Venv {venv} not found.")
>
> Btw, this is the kind of code that makes sense to be inside some
> library.
>
> Thanks,
> Mauro
--
Thanks,
Mauro
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH 05/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (3 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 06/11] docs: parallel-wrapper.sh: remove script Mauro Carvalho Chehab
` (5 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Björn Roy Baron,
Mauro Carvalho Chehab, Alex Gaynor, Alice Ryhl, Andreas Hindborg,
Benno Lossin, Boqun Feng, Danilo Krummrich, Gary Guo,
Miguel Ojeda, Trevor Gross, linux-kernel, rust-for-linux
Now that we have a sphinx-build-wrapper capable of handling
all the needed step to build the supported build targets,
cleanup the Makefile.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 127 ++++++++++-------------------------------
1 file changed, 29 insertions(+), 98 deletions(-)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 2ed334971acd..4013286bef04 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -19,7 +19,6 @@ endif
# You can set these variables from the command line.
SPHINXBUILD = sphinx-build
-SPHINXOPTS =
SPHINXDIRS = .
DOCS_THEME =
DOCS_CSS =
@@ -29,14 +28,14 @@ PAPER =
BUILDDIR = $(obj)/output
PDFLATEX = xelatex
LATEXOPTS = -interaction=batchmode -no-shell-escape
+BUILD_WRAPPER = $(srctree)/scripts/sphinx-build-wrapper
+
+PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
# For denylisting "variable font" files
# Can be overridden by setting as an env variable
FONTS_CONF_DENY_VF ?= $(HOME)/deny-vf
-ifeq ($(findstring 1, $(KBUILD_VERBOSE)),)
-SPHINXOPTS += "-q"
-endif
# User-friendly check for sphinx-build
HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)
@@ -51,62 +50,37 @@ ifeq ($(HAVE_SPHINX),0)
else # HAVE_SPHINX
-# User-friendly check for pdflatex and latexmk
-HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi)
-HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi)
+# Common documentation targets
+infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs:
+ $(Q)@$(srctree)/scripts/sphinx-pre-install --version-check
+ +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
+ --sphinxdirs="$(SPHINXDIRS)" \
+ --conf=$(SPHINX_CONF) \
+ --theme=$(DOCS_THEME) \
+ --css=$(DOCS_CSS) \
+ --paper=$(PAPER)
-ifeq ($(HAVE_LATEXMK),1)
- PDFLATEX := latexmk -$(PDFLATEX)
-endif #HAVE_LATEXMK
-
-# Internal variables.
-PAPEROPT_a4 = -D latex_elements.papersize=a4paper
-PAPEROPT_letter = -D latex_elements.papersize=letterpaper
-ALLSPHINXOPTS = -D kerneldoc_srctree=$(srctree) -D kerneldoc_bin=$(KERNELDOC)
-ALLSPHINXOPTS += $(PAPEROPT_$(PAPER)) $(SPHINXOPTS)
-ifneq ($(wildcard $(srctree)/.config),)
-ifeq ($(CONFIG_RUST),y)
- # Let Sphinx know we will include rustdoc
- ALLSPHINXOPTS += -t rustdoc
-endif
+# Special handling for pdfdocs
+ifeq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0)
+pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
+else
+pdfdocs:
+ $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
+ @echo " SKIP Sphinx $@ target."
endif
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
-
-# commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
-loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
-
-# $2 sphinx builder e.g. "html"
-# $3 name of the build subfolder / e.g. "userspace-api/media", used as:
-# * dest folder relative to $(BUILDDIR) and
-# * cache folder relative to $(BUILDDIR)/.doctrees
-# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man
-# $5 reST source folder relative to $(src),
-# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentation/userspace-api/media
-
-PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
-quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
- cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media $2 && \
- PYTHONPYCACHEPREFIX="$(PYTHONPYCACHEPREFIX)" \
- BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(src)/$5/$(SPHINX_CONF)) \
- $(PYTHON3) $(srctree)/scripts/jobserver-exec \
- $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \
- $(SPHINXBUILD) \
- -b $2 \
- -c $(abspath $(src)) \
- -d $(abspath $(BUILDDIR)/.doctrees/$3) \
- -D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
- $(ALLSPHINXOPTS) \
- $(abspath $(src)/$5) \
- $(abspath $(BUILDDIR)/$3/$4) && \
- if [ "x$(DOCS_CSS)" != "x" ]; then \
- cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(DOCS_CSS)) $(BUILDDIR)/$3/_static/; \
- fi
+infodocs: texinfodocs
+# HTML main logic is identical to other targets. However, if rust is enabled,
+# an extra step at the end is required to generate rustdoc.
htmldocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
+ $(Q)@$(srctree)/scripts/sphinx-pre-install --version-check
+ +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \
+ --sphinxdirs="$(SPHINXDIRS)" \
+ --conf=$(SPHINX_CONF) \
+ --theme=$(DOCS_THEME) \
+ --css=$(DOCS_CSS) \
+ --paper=$(PAPER)
# If Rust support is available and .config exists, add rustdoc generated contents.
# If there are any, the errors from this make rustdoc will be displayed but
@@ -118,49 +92,6 @@ ifeq ($(CONFIG_RUST),y)
endif
endif
-texinfodocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
-
-# Note: the 'info' Make target is generated by sphinx itself when
-# running the texinfodocs target define above.
-infodocs: texinfodocs
- $(MAKE) -C $(BUILDDIR)/texinfo info
-
-linkcheckdocs:
- @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
-
-latexdocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
-
-ifeq ($(HAVE_PDFLATEX),0)
-
-pdfdocs:
- $(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
- @echo " SKIP Sphinx $@ target."
-
-else # HAVE_PDFLATEX
-
-pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
-pdfdocs: latexdocs
- @$(srctree)/scripts/sphinx-pre-install --version-check
- $(foreach var,$(SPHINXDIRS), \
- $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.sh || exit; \
- mkdir -p $(BUILDDIR)/$(var)/pdf; \
- mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUILDDIR)/$(var)/pdf/; \
- )
-
-endif # HAVE_PDFLATEX
-
-epubdocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
-
-xmldocs:
- @$(srctree)/scripts/sphinx-pre-install --version-check
- @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
-
endif # HAVE_SPHINX
# The following targets are independent of HAVE_SPHINX, and the rules should
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 06/11] docs: parallel-wrapper.sh: remove script
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (4 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 05/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 07/11] docs: Makefile: document latex/PDF PAPER= parameter Mauro Carvalho Chehab
` (4 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
The only usage of this script was docs Makefile. Now that
it is using the new sphinx-build-wrapper, which has inside
the code from parallel-wrapper.sh, we can drop this script.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/sphinx/parallel-wrapper.sh | 33 ------------------------
1 file changed, 33 deletions(-)
delete mode 100644 Documentation/sphinx/parallel-wrapper.sh
diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphinx/parallel-wrapper.sh
deleted file mode 100644
index e54c44ce117d..000000000000
--- a/Documentation/sphinx/parallel-wrapper.sh
+++ /dev/null
@@ -1,33 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0+
-#
-# Figure out if we should follow a specific parallelism from the make
-# environment (as exported by scripts/jobserver-exec), or fall back to
-# the "auto" parallelism when "-jN" is not specified at the top-level
-# "make" invocation.
-
-sphinx="$1"
-shift || true
-
-parallel="$PARALLELISM"
-if [ -z "$parallel" ] ; then
- # If no parallelism is specified at the top-level make, then
- # fall back to the expected "-jauto" mode that the "htmldocs"
- # target has had.
- auto=$(perl -e 'open IN,"'"$sphinx"' --version 2>&1 |";
- while (<IN>) {
- if (m/([\d\.]+)/) {
- print "auto" if ($1 >= "1.7")
- }
- }
- close IN')
- if [ -n "$auto" ] ; then
- parallel="$auto"
- fi
-fi
-# Only if some parallelism has been determined do we add the -jN option.
-if [ -n "$parallel" ] ; then
- parallel="-j$parallel"
-fi
-
-exec "$sphinx" $parallel "$@"
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 07/11] docs: Makefile: document latex/PDF PAPER= parameter
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (5 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 06/11] docs: parallel-wrapper.sh: remove script Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 08/11] scripts/sphinx-build-wrapper: restore SPHINXOPTS parsing Mauro Carvalho Chehab
` (3 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
While the build system supports this for a long time, this was
never documented. Add a documentation for it.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 4013286bef04..280728cf78b9 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -129,4 +129,6 @@ dochelp:
@echo
@echo ' make DOCS_CSS={a .css file} adds a DOCS_CSS override file for html/epub output.'
@echo
+ @echo ' make PAPER={a4|letter} Specifies the paper size used for LaTeX/PDF output.'
+ @echo
@echo ' Default location for the generated documents is Documentation/output'
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 08/11] scripts/sphinx-build-wrapper: restore SPHINXOPTS parsing
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (6 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 07/11] docs: Makefile: document latex/PDF PAPER= parameter Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 09/11] scripts: sphinx-build-wrapper: add an argument for LaTeX interactive mode Mauro Carvalho Chehab
` (2 subsequent siblings)
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
While rewriting the wrapper, incidentally support for SPHINXOPTS
was dropped. Restore it and better handle them.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 1 +
scripts/sphinx-build-wrapper | 86 +++++++++++++++++++++++++++++-------
2 files changed, 70 insertions(+), 17 deletions(-)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 280728cf78b9..d2e626627ee6 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -19,6 +19,7 @@ endif
# You can set these variables from the command line.
SPHINXBUILD = sphinx-build
+SPHINXOPTS =
SPHINXDIRS = .
DOCS_THEME =
DOCS_CSS =
diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper
index 5c728956b53c..b9d40f4a6573 100755
--- a/scripts/sphinx-build-wrapper
+++ b/scripts/sphinx-build-wrapper
@@ -148,10 +148,10 @@ class SphinxBuilder:
return path
- def __init__(self, venv=None, verbose=False):
+ def __init__(self, venv=None, verbose=False, n_jobs=None):
"""Initialize internal variables"""
self.venv = venv
- self.verbose = verbose
+ self.verbose = None
# Normal variables passed from Kernel's makefile
self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
@@ -159,6 +159,34 @@ class SphinxBuilder:
self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+ if not verbose:
+ verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
+
+ # Handle SPHINXOPTS evironment
+ sphinxopts = shlex.split(os.environ.get("SPHINXOPTS", ""))
+
+ # As we handle number of jobs and quiet in separate, we need to pick
+ # it the same way as sphinx-build would pick, so let's use argparse
+ # do to the right argument expansion
+ parser = argparse.ArgumentParser()
+ parser.add_argument('-j', '--jobs', type=int)
+ parser.add_argument('-q', '--quiet', type=int)
+
+ # Other sphinx-build arguments go as-is, so place them
+ # at self.sphinxopts
+ sphinx_args, self.sphinxopts = parser.parse_known_args(sphinxopts)
+ if sphinx_args.quiet == True:
+ self.verbose = False
+
+ if sphinx_args.jobs:
+ self.n_jobs = sphinx_args.jobs
+
+ # Command line arguments was passed, override SPHINXOPTS
+ if verbose is not None:
+ self.verbose = verbose
+
+ self.n_jobs = n_jobs
+
# Source tree directory. This needs to be at os.environ, as
# Sphinx extensions and media uAPI makefile needs it
self.srctree = os.environ.get("srctree")
@@ -199,7 +227,7 @@ class SphinxBuilder:
else:
sys.exit(f"Venv {venv} not found.")
- def run_sphinx(self, sphinx_build, sphinx_args, *args, **pwargs):
+ def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
"""
Executes sphinx-build using current python3 command and setting
-j parameter if possible to run the build in parallel.
@@ -207,11 +235,9 @@ class SphinxBuilder:
with JobserverExec() as jobserver:
if jobserver.claim:
- parallelism = str(jobserver.claim)
+ n_jobs = str(jobserver.claim)
else:
- # As Sphinx has parallelism since version 1.7, we don't need
- # any check here.
- parallelism = "auto"
+ n_jobs = "auto" # Supported since Sphinx 1.7
cmd = []
@@ -222,10 +248,19 @@ class SphinxBuilder:
cmd.append(sphinx_build)
- if parallelism:
- cmd.append("-j" + parallelism)
+ # if present, SPHINXOPTS or command line --jobs overrides default
+ if self.n_jobs:
+ n_jobs = str(self.n_jobs)
- cmd += sphinx_args
+ if n_jobs:
+ cmd += [f"-j{n_jobs}"]
+
+ if not self.verbose:
+ cmd.append("-q")
+
+ cmd += self.sphinxopts
+
+ cmd += build_args
if self.verbose:
print(" ".join(cmd))
@@ -447,9 +482,6 @@ class SphinxBuilder:
args.extend(["-D", f"latex_elements.papersize={paper}paper"])
- if not self.verbose:
- args.append("-q")
-
if self.config_rust:
args.extend(["-t", "rustdoc"])
@@ -582,6 +614,25 @@ class SphinxBuilder:
except OSError as e:
sys.exit(f"Failed to restart with {new_python_cmd}: {e}")
+def jobs_type(value):
+ """
+ Handle valid values for -j. Accepts Sphinx "-jauto", plus a number
+ equal or bigger than one.
+ """
+ if value is None:
+ return None
+
+ if value.lower() == 'auto':
+ return value.lower()
+
+ try:
+ if int(value) >= 1:
+ return value
+
+ raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}")
+ except ValueError:
+ raise argparse.ArgumentTypeError(f"Must be 'auto' or positive integer, got {value}")
+
def main():
"""
Main function. The only mandatory argument is the target. If not
@@ -607,18 +658,19 @@ def main():
parser.add_argument("-v", "--verbose", action='store_true',
help="place build in verbose mode")
+ parser.add_argument('-j', '--jobs', type=jobs_type,
+ help="Sets number of jobs to use with sphinx-build")
+
parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
default=None,
help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
args = parser.parse_args()
- if not args.verbose:
- args.verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
-
SphinxBuilder.check_python()
- builder = SphinxBuilder(venv=args.venv, verbose=args.verbose)
+ builder = SphinxBuilder(venv=args.venv, verbose=args.verbose,
+ n_jobs=args.jobs)
builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
theme=args.theme, css=args.css, paper=args.paper)
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 09/11] scripts: sphinx-build-wrapper: add an argument for LaTeX interactive mode
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (7 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 08/11] scripts/sphinx-build-wrapper: restore SPHINXOPTS parsing Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 10/11] scripts: sphinx-*: prevent sphinx-build crashes Mauro Carvalho Chehab
2025-08-15 11:50 ` [PATCH 11/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper Mauro Carvalho Chehab
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
By default, we use LaTeX batch mode to build docs. This way, when
an error happens, the build fails. This is good for normal builds,
but when debugging problems with pdf generation, the best is to
use interactive mode.
We already support it via LATEXOPTS, but having a command line
argument makes it easier:
Interactive mode:
./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v -i
...
Running 'xelatex --no-pdf -no-pdf -recorder ".../Documentation/output/peci/latex/peci.tex"'
...
Default batch mode:
./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v
...
Running 'xelatex --no-pdf -no-pdf -interaction=batchmode -no-shell-escape -recorder ".../Documentation/output/peci/latex/peci.tex"'
...
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/sphinx-build-wrapper | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper
index b9d40f4a6573..f6fec766c3e6 100755
--- a/scripts/sphinx-build-wrapper
+++ b/scripts/sphinx-build-wrapper
@@ -148,7 +148,7 @@ class SphinxBuilder:
return path
- def __init__(self, venv=None, verbose=False, n_jobs=None):
+ def __init__(self, venv=None, verbose=False, n_jobs=None, interactive=None):
"""Initialize internal variables"""
self.venv = venv
self.verbose = None
@@ -157,7 +157,11 @@ class SphinxBuilder:
self.kernelversion = os.environ.get("KERNELVERSION", "unknown")
self.kernelrelease = os.environ.get("KERNELRELEASE", "unknown")
self.pdflatex = os.environ.get("PDFLATEX", "xelatex")
- self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+
+ if not interactive:
+ self.latexopts = os.environ.get("LATEXOPTS", "-interaction=batchmode -no-shell-escape")
+ else:
+ self.latexopts = os.environ.get("LATEXOPTS", "")
if not verbose:
verbose = bool(os.environ.get("KBUILD_VERBOSE", "") != "")
@@ -661,6 +665,9 @@ def main():
parser.add_argument('-j', '--jobs', type=jobs_type,
help="Sets number of jobs to use with sphinx-build")
+ parser.add_argument('-i', '--interactive', action='store_true',
+ help="Change latex default to run in interactive mode")
+
parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
default=None,
help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
@@ -670,7 +677,7 @@ def main():
SphinxBuilder.check_python()
builder = SphinxBuilder(venv=args.venv, verbose=args.verbose,
- n_jobs=args.jobs)
+ n_jobs=args.jobs, interactive=args.interactive)
builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
theme=args.theme, css=args.css, paper=args.paper)
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 10/11] scripts: sphinx-*: prevent sphinx-build crashes
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (8 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 09/11] scripts: sphinx-build-wrapper: add an argument for LaTeX interactive mode Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
2025-08-21 19:41 ` Jonathan Corbet
2025-08-15 11:50 ` [PATCH 11/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper Mauro Carvalho Chehab
10 siblings, 1 reply; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
On a properly set system, LANG and LC_ALL is always defined.
However, some distros like Debian, Gentoo and their variants
start with those undefioned.
When Sphinx tries to set a locale with:
locale.setlocale(locale.LC_ALL, '')
It raises an exception, making Sphinx fail. This is more likely
to happen with test containers.
Add a logic to detect and workaround such issue by setting
locale to C.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
scripts/sphinx-build-wrapper | 10 ++++++++++
scripts/sphinx-pre-install | 14 +++++++++++++-
2 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper
index f6fec766c3e6..f21701d34552 100755
--- a/scripts/sphinx-build-wrapper
+++ b/scripts/sphinx-build-wrapper
@@ -45,6 +45,7 @@ the newer version.
"""
import argparse
+import locale
import os
import re
import shlex
@@ -495,6 +496,15 @@ class SphinxBuilder:
if not sphinxdirs:
sphinxdirs = os.environ.get("SPHINXDIRS", ".")
+ # The sphinx-build tool has a bug: internally, it tries to set
+ # locale with locale.setlocale(locale.LC_ALL, ''). This causes a
+ # crash if language is not set. Detect and fix it.
+ try:
+ locale.setlocale(locale.LC_ALL, '')
+ except Exception:
+ self.env["LC_ALL"] = "C"
+ self.env["LANG"] = "C"
+
# sphinxdirs can be a list or a whitespace-separated string
sphinxdirs_list = []
for sphinxdir in sphinxdirs:
diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
index c46d7b76f93c..50ae9cfcb3d7 100755
--- a/scripts/sphinx-pre-install
+++ b/scripts/sphinx-pre-install
@@ -26,6 +26,7 @@ system pacage install is recommended.
"""
import argparse
+import locale
import os
import re
import subprocess
@@ -516,8 +517,19 @@ class MissingCheckers(AncillaryMethods):
"""
Gets sphinx-build version.
"""
+ env = os.environ.copy()
+
+ # The sphinx-build tool has a bug: internally, it tries to set
+ # locale with locale.setlocale(locale.LC_ALL, ''). This causes a
+ # crash if language is not set. Detect and fix it.
try:
- result = self.run([cmd, "--version"],
+ locale.setlocale(locale.LC_ALL, '')
+ except Exception:
+ env["LC_ALL"] = "C"
+ env["LANG"] = "C"
+
+ try:
+ result = self.run([cmd, "--version"], env=env,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True, check=True)
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread* Re: [PATCH 10/11] scripts: sphinx-*: prevent sphinx-build crashes
2025-08-15 11:50 ` [PATCH 10/11] scripts: sphinx-*: prevent sphinx-build crashes Mauro Carvalho Chehab
@ 2025-08-21 19:41 ` Jonathan Corbet
0 siblings, 0 replies; 21+ messages in thread
From: Jonathan Corbet @ 2025-08-21 19:41 UTC (permalink / raw)
To: Mauro Carvalho Chehab, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> + # The sphinx-build tool has a bug: internally, it tries to set
> + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a
> + # crash if language is not set. Detect and fix it.
> + try:
> + locale.setlocale(locale.LC_ALL, '')
> + except Exception:
> + self.env["LC_ALL"] = "C"
> + self.env["LANG"] = "C"
In my years of writing Python, one of the most insidious things I have
encountered is this sort of wild-card catch; it can hide all kinds of
bugs. There must be a specific exception you can catch here?
Thanks,
jon
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH 11/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper
2025-08-15 11:50 [PATCH 00/11] Split sphinx call logic from docs Makefile Mauro Carvalho Chehab
` (9 preceding siblings ...)
2025-08-15 11:50 ` [PATCH 10/11] scripts: sphinx-*: prevent sphinx-build crashes Mauro Carvalho Chehab
@ 2025-08-15 11:50 ` Mauro Carvalho Chehab
10 siblings, 0 replies; 21+ messages in thread
From: Mauro Carvalho Chehab @ 2025-08-15 11:50 UTC (permalink / raw)
To: Jonathan Corbet, Linux Doc Mailing List
Cc: Mauro Carvalho Chehab, Mauro Carvalho Chehab, linux-kernel
Now that we have a sphinx-build-wrapper capable of handling
all the needed step to build the supported build targets,
cleanup the Makefile.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
Documentation/Makefile | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/Documentation/Makefile b/Documentation/Makefile
index d2e626627ee6..5a3dc9e5b578 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -19,11 +19,9 @@ endif
# You can set these variables from the command line.
SPHINXBUILD = sphinx-build
-SPHINXOPTS =
SPHINXDIRS = .
DOCS_THEME =
DOCS_CSS =
-_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
SPHINX_CONF = conf.py
PAPER =
BUILDDIR = $(obj)/output
@@ -33,6 +31,12 @@ BUILD_WRAPPER = $(srctree)/scripts/sphinx-build-wrapper
PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
+PYTHONPYCACHEPREFIX ?= $(abspath $(BUILDDIR)/__pycache__)
+
+# Wrapper for sphinx-build
+
+BUILD_WRAPPER = $(srctree)/scripts/sphinx-build-wrapper
+
# For denylisting "variable font" files
# Can be overridden by setting as an env variable
FONTS_CONF_DENY_VF ?= $(HOME)/deny-vf
@@ -103,7 +107,9 @@ refcheckdocs:
cleandocs:
$(Q)rm -rf $(BUILDDIR)
- $(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media clean
+
+# Used only on help
+_SPHINXDIRS = $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%,$(wildcard $(srctree)/Documentation/*/index.rst)))
dochelp:
@echo ' Linux kernel internal documentation in different formats from ReST:'
--
2.50.1
^ permalink raw reply related [flat|nested] 21+ messages in thread