Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula <jani.nikula@linux.intel.com>]

On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>
>> Because we want reproducible builds
>> and https://github.com/sphinx-doc/sphinx/issues/6714
>> did not receive any love from Sphinx devs in five years,
>> let's disable parallel doc builds until that Sphinx issue is fixed.
>
> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
> Sphinx instances running in parallel and racing in doctree access.
>
> In kernel, does the issue then boil down to:
>
> htmldocs:
> 	@$(srctree)/scripts/sphinx-pre-install --version-check
> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>
> i.e. multiple Sphinx invocations instead of just one?
>
> Broken record, I still think we should axe the Makefile hacks out of the
> Sphinx build altogether, and let Sphinx handle everything, via
> extensions if necessary.

Indeed. On my machine, 'make htmldocs' spawns eight sphinx-build
processes, each running with -j8, and each racing for the same
doctrees. And the whole thing is apparently parallelized to 64 threads,
which is unlikely to be the most efficient.

There's no reason to blame Sphinx upstream before we get our end
together, and run Sphinx as it was designed.


BR,
Jani.


>
> BR,
> Jani.
>
>
> [1] https://github.com/sphinx-doc/sphinx/issues/6714#issuecomment-1848975385
> [2] https://github.com/sphinx-doc/sphinx/issues/2946
>
>>
>> This patch was done while working on reproducible builds for openSUSE,
>> sponsored by the NLnet NGI0 fund.
>>
>> Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
>> ---
>>  Documentation/sphinx/parallel-wrapper.sh | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphinx/parallel-wrapper.sh
>> index e54c44ce117d..cb93626bd86e 100644
>> --- a/Documentation/sphinx/parallel-wrapper.sh
>> +++ b/Documentation/sphinx/parallel-wrapper.sh
>> @@ -10,6 +10,9 @@ sphinx="$1"
>>  shift || true
>>  
>>  parallel="$PARALLELISM"
>> +# Because of issues in Sphinx(https://github.com/sphinx-doc/sphinx/issues/6714)
>> +# we disable parallel doc generation to get deterministic build results
>> +parallel=1
>>  if [ -z "$parallel" ] ; then
>>  	# If no parallelism is specified at the top-level make, then
>>  	# fall back to the expected "-jauto" mode that the "htmldocs"

-- 
Jani Nikula, Intel