Re: [PATCH v4 08/19] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build

[Jani Nikula <jani.nikula@linux.intel.com>]

On Wed, 10 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Wed, 10 Sep 2025 13:46:17 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
>> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> > There are too much magic inside docs Makefile to properly run
>> > sphinx-build. Create an ancillary script that contains all
>> > kernel-related sphinx-build call logic currently at Makefile.
>> >
>> > Such script is designed to work both as an standalone command
>> > and as part of a Makefile. As such, it properly handles POSIX
>> > jobserver used by GNU make.
>> >
>> > On a side note, there was a line number increase due to the
>> > conversion:
>> >
>> >  Documentation/Makefile          |  131 +++----------
>> >  tools/docs/sphinx-build-wrapper |  293 +++++++++++++++++++++++++++++++
>> >  2 files changed, 323 insertions(+), 101 deletions(-)
>> >
>> > This is because some things are more verbosed on Python and because
>> > it requires reading env vars from Makefile. Besides it, this script
>> > has some extra features that don't exist at the Makefile:
>> >
>> > - It can be called directly from command line;
>> > - It properly return PDF build errors.
>> >
>> > When running the script alone, it will only take handle sphinx-build
>> > targets. On other words, it won't runn make rustdoc after building
>> > htmlfiles, nor it will run the extra check scripts.  
>> 
>> I've always strongly believed we should aim to make it possible to build
>> the documentation by running sphinx-build directly on the
>> command-line. Not that it would be the common way to run it, but to not
>> accumulate things in the Makefile that need to happen before or
>> after. To promote handling the documentation build in Sphinx. To be able
>> to debug issues and try new Sphinx versions without all the hacks.
>
> That would be the better, but, unfortunately, this is not possible, for 
> several reasons:
>
> 1. SPHINXDIRS. It needs a lot of magic to work, both before running
>    sphinx-build and after (inside conf.py);

Makes you wonder if that's the right solution to the original
problem. It was added as a kind of hack, and it stuck.

> 2. Several extensions require kernel-specific environment variables to
>    work. Calling sphinx-build directly breaks them;

The extensions shouldn't be using environment variables for
configuration anyway. Add config options and set them in conf.py like
everything else?

> 3. Sphinx itself doesn't build several targets alone. Instead, they create
>    a Makefile, and an extra step is needed to finish the build. That's 
>    the case for pdf and texinfo, for instance;

That's not true for the Makefile currently generated by
sphinx-quickstart. Granted, I haven't used Sphinx much for pdf output.

> 4. Man pages generation. Sphinx support to generate it is very poor;

In what way?

> 5. Rust integration adds more complexity to the table;
>
> I'm not seeing sphinx-build supporting the above needs anytime soon,
> and, even if we push our needs to Sphinx and it gets accepted there,
> we'll still need to wait for quite a while until LTS distros merge
> them.

I'm not suggesting to add anything to Sphinx upstream.

>> This patch moves a bunch of that logic into a Python wrapper, and I feel
>> like it complicates matters. You can no longer rely on 'make V=1' to get
>> the build commands, for instance.
>
> Quite the opposite. if you try using "make V=1", it won't show the
> command line used to call sphinx-build anymore.
>
> This series restore it.
>
> See, if you build with this series with V=1, you will see exactly
> what commands are used on the build:
>
> 	$ make V=1 htmldocs
> 	...
> 	python3 ./tools/docs/sphinx-build-wrapper htmldocs \
> 	        --sphinxdirs="." --conf="conf.py" \
>         	--builddir="Documentation/output" \
> 	        --theme= --css= --paper=
> 	python3 /new_devel/docs/sphinx_latest/bin/sphinx-build -j25 -b html -c /new_devel/docs/Documentation -d /new_devel/docs/Documentation/output/.doctrees -D kerneldoc_bin=scripts/kernel-doc.py -D version=6.17.0-rc1 -D release=6.17.0-rc1+ -D kerneldoc_srctree=. /new_devel/docs/Documentation /new_devel/docs/Documentation/output
> 	...
>
>
>
>> Newer Sphinx versions have the -M option for "make mode". The Makefiles
>> produced by sphinx-quickstart only have one build target:
>> 
>> # Catch-all target: route all unknown targets to Sphinx using the new
>> # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
>
> I didn't know about this, but from [1] it sounds it covers just two
> targets: "latexpdf" and "info".

sphinx-build -M help gives a list of 24 targets.

> The most complex scenario is still not covered: SPHINXDIRS.
>
> [1] https://www.sphinx-doc.org/en/master/man/sphinx-build.html
>
>> %: Makefile
>>         @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
>> 
>> That's all.
>
> Try doing such change on your makefile. it will break:
>
> 	- SPHINXDIRS;
> 	- V=1;
> 	- rustdoc

I know it does. That's the problem.

> and will still be dependent on variables that are passed via
> env from Kernel makefile. So, stil you can't run from command
> line. Also, if you call sphinx-build from command line:
>
> 	$ sphinx-build -j25 -b html Documentation Documentation/output
> 	...
> 	      File "<frozen os>", line 717, in __getitem__
> 	    KeyError: 'srctree'
>
> It won't work, as several parameters that are required by conf.py and by
> Sphinx extensions would be missing (the most important one is srctree, but
> there are others in the line too).
>
>> The proposed wrapper duplicates loads of code that's supposed to be
>> handled by sphinx-build directly.
>
> Once we get the wrapper, we can work to simplify it, but still I
> can't see how to get rid of it.

I just don't understand the mentality of first adding something complex,
and then working to simplify it.

Don't make it a Rube Goldberg machine in the first place.

>> Including the target/builder names.
>
> True, but this was a design decision taken lots of years ago: instead
> of:
> 	make html
>
> we're using:
>
> 	make htmldocs
>
> This series doesn't change that: either makefile or the script need
> to tho the namespace conversion.

In the above Makefile snippet that conversion would be $(@:docs=)

The clean Makefile way of checking for having Sphinx and the required
versions of Python and dependencies etc. would be a .PHONY target that
just checks, and doesn't do *anything* else. It shouldn't be part of the
sphinx-build rules.

PHONY += check-versions
check-versions:
	sphinx-pre-install --version-check

htmldocs: check-versions
	...

Or something like that.

>> Seems to me the goal should be to figure out *generic* wrappers for
>> handling parallelism, not Sphinx aware/specific.
>> 
>> 
>> BR,
>> Jani.

-- 
Jani Nikula, Intel