Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula <jani.nikula@linux.intel.com>]

On Thu, 05 Sep 2024, Vegard Nossum <vegard.nossum@oracle.com> wrote:
> On 05/09/2024 14:20, Jani Nikula wrote:
>> On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>>> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>>>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>>>
>>>> Because we want reproducible builds
>>>> and https://github.com/sphinx-doc/sphinx/issues/6714
>>>> did not receive any love from Sphinx devs in five years,
>>>> let's disable parallel doc builds until that Sphinx issue is fixed.
>>>
>>> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
>>> Sphinx instances running in parallel and racing in doctree access.
>>>
>>> In kernel, does the issue then boil down to:
>>>
>>> htmldocs:
>>> 	@$(srctree)/scripts/sphinx-pre-install --version-check
>>> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>>>
>>> i.e. multiple Sphinx invocations instead of just one?
>>>
>>> Broken record, I still think we should axe the Makefile hacks out of the
>>> Sphinx build altogether, and let Sphinx handle everything, via
>>> extensions if necessary.
>> 
>> Indeed. On my machine, 'make htmldocs' spawns eight sphinx-build
>> processes, each running with -j8, and each racing for the same
>> doctrees. And the whole thing is apparently parallelized to 64 threads,
>> which is unlikely to be the most efficient.
>> 
>> There's no reason to blame Sphinx upstream before we get our end
>> together, and run Sphinx as it was designed.
>
> I don't reproduce this here, either with plain 'make htmldocs' or 'make
> -j8 htmldocs' -- in either case the process tree is:
>
> make ... htmldocs
> - make ... htmldocs
>    - make -f ./scripts/Makefile.build obj=Documentation htmldocs
>      - /bin/sh -c make ...
>        - python3 ./scripts/jobserver-exec sh
> ./Documentation/sphinx/parallel-wrapper.sh sphinx-build -b html ...
>          - /usr/bin/python3 /usr/bin/sphinx-build -j... -b html ...
>
> Neither the $(foreach) in the Makefile nor $(call loop_cmd) should
> result in parallel invocations.
>
> Could you double check the process tree once the "Using alabaster theme"
> message has appeared?

$ pstree -A 3457144
bash---make---make---make---make---sh---python3---sphinx-build-+-7*[sphinx-build]
                                                               `-sphinx-build---perl

Maybe try after 'make cleandocs'.

BR,
Jani.

>
>
> Vegard

-- 
Jani Nikula, Intel