[PATCH] docs: Build kernel docs deterministically

[PATCH] docs: Build kernel docs deterministically

[bernhard+linux-doc]

From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>

Because we want reproducible builds
and https://github.com/sphinx-doc/sphinx/issues/6714
did not receive any love from Sphinx devs in five years,
let's disable parallel doc builds until that Sphinx issue is fixed.

This patch was done while working on reproducible builds for openSUSE,
sponsored by the NLnet NGI0 fund.

Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
---
 Documentation/sphinx/parallel-wrapper.sh | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphinx/parallel-wrapper.sh
index e54c44ce117d..cb93626bd86e 100644
--- a/Documentation/sphinx/parallel-wrapper.sh
+++ b/Documentation/sphinx/parallel-wrapper.sh
@@ -10,6 +10,9 @@ sphinx="$1"
 shift || true
 
 parallel="$PARALLELISM"
+# Because of issues in Sphinx(https://github.com/sphinx-doc/sphinx/issues/6714)
+# we disable parallel doc generation to get deterministic build results
+parallel=1
 if [ -z "$parallel" ] ; then
 	# If no parallelism is specified at the top-level make, then
 	# fall back to the expected "-jauto" mode that the "htmldocs"
-- 
2.46.0

Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula]

On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>
> Because we want reproducible builds
> and https://github.com/sphinx-doc/sphinx/issues/6714
> did not receive any love from Sphinx devs in five years,
> let's disable parallel doc builds until that Sphinx issue is fixed.

You mention in [1] that this is likely a duplicate of [2] i.e. multiple
Sphinx instances running in parallel and racing in doctree access.

In kernel, does the issue then boil down to:

htmldocs:
	@$(srctree)/scripts/sphinx-pre-install --version-check
	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))

i.e. multiple Sphinx invocations instead of just one?

Broken record, I still think we should axe the Makefile hacks out of the
Sphinx build altogether, and let Sphinx handle everything, via
extensions if necessary.

BR,
Jani.


[1] https://github.com/sphinx-doc/sphinx/issues/6714#issuecomment-1848975385
[2] https://github.com/sphinx-doc/sphinx/issues/2946

>
> This patch was done while working on reproducible builds for openSUSE,
> sponsored by the NLnet NGI0 fund.
>
> Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
> ---
>  Documentation/sphinx/parallel-wrapper.sh | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphinx/parallel-wrapper.sh
> index e54c44ce117d..cb93626bd86e 100644
> --- a/Documentation/sphinx/parallel-wrapper.sh
> +++ b/Documentation/sphinx/parallel-wrapper.sh
> @@ -10,6 +10,9 @@ sphinx="$1"
>  shift || true
>  
>  parallel="$PARALLELISM"
> +# Because of issues in Sphinx(https://github.com/sphinx-doc/sphinx/issues/6714)
> +# we disable parallel doc generation to get deterministic build results
> +parallel=1
>  if [ -z "$parallel" ] ; then
>  	# If no parallelism is specified at the top-level make, then
>  	# fall back to the expected "-jauto" mode that the "htmldocs"

-- 
Jani Nikula, Intel

Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula]

On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>
>> Because we want reproducible builds
>> and https://github.com/sphinx-doc/sphinx/issues/6714
>> did not receive any love from Sphinx devs in five years,
>> let's disable parallel doc builds until that Sphinx issue is fixed.
>
> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
> Sphinx instances running in parallel and racing in doctree access.
>
> In kernel, does the issue then boil down to:
>
> htmldocs:
> 	@$(srctree)/scripts/sphinx-pre-install --version-check
> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>
> i.e. multiple Sphinx invocations instead of just one?
>
> Broken record, I still think we should axe the Makefile hacks out of the
> Sphinx build altogether, and let Sphinx handle everything, via
> extensions if necessary.

Indeed. On my machine, 'make htmldocs' spawns eight sphinx-build
processes, each running with -j8, and each racing for the same
doctrees. And the whole thing is apparently parallelized to 64 threads,
which is unlikely to be the most efficient.

There's no reason to blame Sphinx upstream before we get our end
together, and run Sphinx as it was designed.


BR,
Jani.


>
> BR,
> Jani.
>
>
> [1] https://github.com/sphinx-doc/sphinx/issues/6714#issuecomment-1848975385
> [2] https://github.com/sphinx-doc/sphinx/issues/2946
>
>>
>> This patch was done while working on reproducible builds for openSUSE,
>> sponsored by the NLnet NGI0 fund.
>>
>> Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
>> ---
>>  Documentation/sphinx/parallel-wrapper.sh | 3 +++
>>  1 file changed, 3 insertions(+)
>>
>> diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphinx/parallel-wrapper.sh
>> index e54c44ce117d..cb93626bd86e 100644
>> --- a/Documentation/sphinx/parallel-wrapper.sh
>> +++ b/Documentation/sphinx/parallel-wrapper.sh
>> @@ -10,6 +10,9 @@ sphinx="$1"
>>  shift || true
>>  
>>  parallel="$PARALLELISM"
>> +# Because of issues in Sphinx(https://github.com/sphinx-doc/sphinx/issues/6714)
>> +# we disable parallel doc generation to get deterministic build results
>> +parallel=1
>>  if [ -z "$parallel" ] ; then
>>  	# If no parallelism is specified at the top-level make, then
>>  	# fall back to the expected "-jauto" mode that the "htmldocs"

-- 
Jani Nikula, Intel

Re: [PATCH] docs: Build kernel docs deterministically

[Bernhard M. Wiedemann]

On 05/09/2024 14.04, Jani Nikula wrote:
> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>
>> Because we want reproducible builds
>> and https://github.com/sphinx-doc/sphinx/issues/6714
>> did not receive any love from Sphinx devs in five years,
>> let's disable parallel doc builds until that Sphinx issue is fixed.
> 
> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
> Sphinx instances running in parallel and racing in doctree access.
> 
> In kernel, does the issue then boil down to:
> 
> htmldocs:
> 	@$(srctree)/scripts/sphinx-pre-install --version-check
> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
> 
> i.e. multiple Sphinx invocations instead of just one?

If that is the case, then providing a unique doctree dir to each 
invocation should also help.

However my patch for sphinx -j1 did give good test results, too.
Maybe in your case that would result in 8 sphinx calls with 1 thread 
each, which would be more appropriate for your machine.

Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula]

On Thu, 05 Sep 2024, "Bernhard M. Wiedemann" <bwiedemann@suse.de> wrote:
> On 05/09/2024 14.04, Jani Nikula wrote:
>> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>>
>>> Because we want reproducible builds
>>> and https://github.com/sphinx-doc/sphinx/issues/6714
>>> did not receive any love from Sphinx devs in five years,
>>> let's disable parallel doc builds until that Sphinx issue is fixed.
>> 
>> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
>> Sphinx instances running in parallel and racing in doctree access.
>> 
>> In kernel, does the issue then boil down to:
>> 
>> htmldocs:
>> 	@$(srctree)/scripts/sphinx-pre-install --version-check
>> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>> 
>> i.e. multiple Sphinx invocations instead of just one?
>
> If that is the case, then providing a unique doctree dir to each 
> invocation should also help.

I'm not sure that's a good idea, because IIUC there should be one
doctree.

> However my patch for sphinx -j1 did give good test results, too.
> Maybe in your case that would result in 8 sphinx calls with 1 thread 
> each, which would be more appropriate for your machine.

The right thing to do is to have one sphinx-build process and pass -j<N>
to that.

BR,
Jani.

-- 
Jani Nikula, Intel

Re: [PATCH] docs: Build kernel docs deterministically

[Vegard Nossum]

On 05/09/2024 14:20, Jani Nikula wrote:
> On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>>
>>> Because we want reproducible builds
>>> and https://github.com/sphinx-doc/sphinx/issues/6714
>>> did not receive any love from Sphinx devs in five years,
>>> let's disable parallel doc builds until that Sphinx issue is fixed.
>>
>> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
>> Sphinx instances running in parallel and racing in doctree access.
>>
>> In kernel, does the issue then boil down to:
>>
>> htmldocs:
>> 	@$(srctree)/scripts/sphinx-pre-install --version-check
>> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>>
>> i.e. multiple Sphinx invocations instead of just one?
>>
>> Broken record, I still think we should axe the Makefile hacks out of the
>> Sphinx build altogether, and let Sphinx handle everything, via
>> extensions if necessary.
> 
> Indeed. On my machine, 'make htmldocs' spawns eight sphinx-build
> processes, each running with -j8, and each racing for the same
> doctrees. And the whole thing is apparently parallelized to 64 threads,
> which is unlikely to be the most efficient.
> 
> There's no reason to blame Sphinx upstream before we get our end
> together, and run Sphinx as it was designed.

I don't reproduce this here, either with plain 'make htmldocs' or 'make
-j8 htmldocs' -- in either case the process tree is:

make ... htmldocs
- make ... htmldocs
   - make -f ./scripts/Makefile.build obj=Documentation htmldocs
     - /bin/sh -c make ...
       - python3 ./scripts/jobserver-exec sh
./Documentation/sphinx/parallel-wrapper.sh sphinx-build -b html ...
         - /usr/bin/python3 /usr/bin/sphinx-build -j... -b html ...

Neither the $(foreach) in the Makefile nor $(call loop_cmd) should
result in parallel invocations.

Could you double check the process tree once the "Using alabaster theme"
message has appeared?


Vegard

Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula]

On Thu, 05 Sep 2024, Vegard Nossum <vegard.nossum@oracle.com> wrote:
> On 05/09/2024 14:20, Jani Nikula wrote:
>> On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>>> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>>>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>>>
>>>> Because we want reproducible builds
>>>> and https://github.com/sphinx-doc/sphinx/issues/6714
>>>> did not receive any love from Sphinx devs in five years,
>>>> let's disable parallel doc builds until that Sphinx issue is fixed.
>>>
>>> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
>>> Sphinx instances running in parallel and racing in doctree access.
>>>
>>> In kernel, does the issue then boil down to:
>>>
>>> htmldocs:
>>> 	@$(srctree)/scripts/sphinx-pre-install --version-check
>>> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>>>
>>> i.e. multiple Sphinx invocations instead of just one?
>>>
>>> Broken record, I still think we should axe the Makefile hacks out of the
>>> Sphinx build altogether, and let Sphinx handle everything, via
>>> extensions if necessary.
>> 
>> Indeed. On my machine, 'make htmldocs' spawns eight sphinx-build
>> processes, each running with -j8, and each racing for the same
>> doctrees. And the whole thing is apparently parallelized to 64 threads,
>> which is unlikely to be the most efficient.
>> 
>> There's no reason to blame Sphinx upstream before we get our end
>> together, and run Sphinx as it was designed.
>
> I don't reproduce this here, either with plain 'make htmldocs' or 'make
> -j8 htmldocs' -- in either case the process tree is:
>
> make ... htmldocs
> - make ... htmldocs
>    - make -f ./scripts/Makefile.build obj=Documentation htmldocs
>      - /bin/sh -c make ...
>        - python3 ./scripts/jobserver-exec sh
> ./Documentation/sphinx/parallel-wrapper.sh sphinx-build -b html ...
>          - /usr/bin/python3 /usr/bin/sphinx-build -j... -b html ...
>
> Neither the $(foreach) in the Makefile nor $(call loop_cmd) should
> result in parallel invocations.
>
> Could you double check the process tree once the "Using alabaster theme"
> message has appeared?

$ pstree -A 3457144
bash---make---make---make---make---sh---python3---sphinx-build-+-7*[sphinx-build]
                                                               `-sphinx-build---perl

Maybe try after 'make cleandocs'.

BR,
Jani.

>
>
> Vegard

-- 
Jani Nikula, Intel

Re: [PATCH] docs: Build kernel docs deterministically

[Vegard Nossum]

On 05/09/2024 16:08, Jani Nikula wrote:
> On Thu, 05 Sep 2024, Vegard Nossum <vegard.nossum@oracle.com> wrote:
>> On 05/09/2024 14:20, Jani Nikula wrote:
>>> On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>>>> On Thu, 05 Sep 2024, bernhard+linux-doc@lsmod.de wrote:
>>>>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>>>>>
>>>>> Because we want reproducible builds
>>>>> and https://github.com/sphinx-doc/sphinx/issues/6714
>>>>> did not receive any love from Sphinx devs in five years,
>>>>> let's disable parallel doc builds until that Sphinx issue is fixed.
>>>>
>>>> You mention in [1] that this is likely a duplicate of [2] i.e. multiple
>>>> Sphinx instances running in parallel and racing in doctree access.
>>>>
>>>> In kernel, does the issue then boil down to:
>>>>
>>>> htmldocs:
>>>> 	@$(srctree)/scripts/sphinx-pre-install --version-check
>>>> 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>>>>
>>>> i.e. multiple Sphinx invocations instead of just one?
>>>>
>>>> Broken record, I still think we should axe the Makefile hacks out of the
>>>> Sphinx build altogether, and let Sphinx handle everything, via
>>>> extensions if necessary.
>>>
>>> Indeed. On my machine, 'make htmldocs' spawns eight sphinx-build
>>> processes, each running with -j8, and each racing for the same
>>> doctrees. And the whole thing is apparently parallelized to 64 threads,
>>> which is unlikely to be the most efficient.
>>>
>>> There's no reason to blame Sphinx upstream before we get our end
>>> together, and run Sphinx as it was designed.
>>
>> I don't reproduce this here, either with plain 'make htmldocs' or 'make
>> -j8 htmldocs' -- in either case the process tree is:
>>
>> make ... htmldocs
>> - make ... htmldocs
>>     - make -f ./scripts/Makefile.build obj=Documentation htmldocs
>>       - /bin/sh -c make ...
>>         - python3 ./scripts/jobserver-exec sh
>> ./Documentation/sphinx/parallel-wrapper.sh sphinx-build -b html ...
>>           - /usr/bin/python3 /usr/bin/sphinx-build -j... -b html ...
>>
>> Neither the $(foreach) in the Makefile nor $(call loop_cmd) should
>> result in parallel invocations.
>>
>> Could you double check the process tree once the "Using alabaster theme"
>> message has appeared?
> 
> $ pstree -A 3457144
> bash---make---make---make---make---sh---python3---sphinx-build-+-7*[sphinx-build]
>                                                                 `-sphinx-build---perl
> 
> Maybe try after 'make cleandocs'.

Right, I get this with make -j8 after 'make cleandocs':

make---make---make---sh---python3---sphinx-build-+-sphinx-build---perl
                                                  `-7*[sphinx-build]

Isn't this exactly what we want? There is only a single top-level
sphinx-build process invoked by 'make', and that is running 8 processes
in parallel, not 64 threads.

I think -j8 appear in these child processes simply because it inherited
the command line arguments (so fork() without exec(), essentially).


Vegard

Re: [PATCH] docs: Build kernel docs deterministically

[Jonathan Corbet]

Jani Nikula <jani.nikula@linux.intel.com> writes:

>> However my patch for sphinx -j1 did give good test results, too.
>> Maybe in your case that would result in 8 sphinx calls with 1 thread 
>> each, which would be more appropriate for your machine.
>
> The right thing to do is to have one sphinx-build process and pass -j<N>
> to that.

[I wouldn't have minded being CC'd on this conversation...]

I, too, have never seen the behavior Jani reports.

I, too, would like to get rid of as much of the makefile hackery as
possible, but it all did end up there for a reason.

The business around parallelism was intended to make sphinx play well
with other targets being built in the same make invocation.  If you do a
"make -j8 this that theother htmldocs" and there are five processes
working on this, that, and theother, then sphinx should not create more
than three.  See 51e46c7a4007 for more.

In Jani's case, it sounds like the job-slot reservation isn't working
right somehow?

jon

Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula]

On Thu, 05 Sep 2024, Jonathan Corbet <corbet@lwn.net> wrote:
> Jani Nikula <jani.nikula@linux.intel.com> writes:
>
>>> However my patch for sphinx -j1 did give good test results, too.
>>> Maybe in your case that would result in 8 sphinx calls with 1 thread 
>>> each, which would be more appropriate for your machine.
>>
>> The right thing to do is to have one sphinx-build process and pass -j<N>
>> to that.
>
> [I wouldn't have minded being CC'd on this conversation...]

Yes, sorry about that.

> I, too, have never seen the behavior Jani reports.
>
> I, too, would like to get rid of as much of the makefile hackery as
> possible, but it all did end up there for a reason.
>
> The business around parallelism was intended to make sphinx play well
> with other targets being built in the same make invocation.  If you do a
> "make -j8 this that theother htmldocs" and there are five processes
> working on this, that, and theother, then sphinx should not create more
> than three.  See 51e46c7a4007 for more.
>
> In Jani's case, it sounds like the job-slot reservation isn't working
> right somehow?

Looking at the -j<N> results on an unrelated project, PEBKAC is a
distinct possibility here, and this part may be a red herring. I'll need
to look into it.

BR,
Jani.


-- 
Jani Nikula, Intel

Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula]

On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
> On Thu, 05 Sep 2024, Jonathan Corbet <corbet@lwn.net> wrote:
>> Jani Nikula <jani.nikula@linux.intel.com> writes:
>>
>>>> However my patch for sphinx -j1 did give good test results, too.
>>>> Maybe in your case that would result in 8 sphinx calls with 1 thread 
>>>> each, which would be more appropriate for your machine.
>>>
>>> The right thing to do is to have one sphinx-build process and pass -j<N>
>>> to that.
>>
>> [I wouldn't have minded being CC'd on this conversation...]
>
> Yes, sorry about that.
>
>> I, too, have never seen the behavior Jani reports.
>>
>> I, too, would like to get rid of as much of the makefile hackery as
>> possible, but it all did end up there for a reason.
>>
>> The business around parallelism was intended to make sphinx play well
>> with other targets being built in the same make invocation.  If you do a
>> "make -j8 this that theother htmldocs" and there are five processes
>> working on this, that, and theother, then sphinx should not create more
>> than three.  See 51e46c7a4007 for more.
>>
>> In Jani's case, it sounds like the job-slot reservation isn't working
>> right somehow?
>
> Looking at the -j<N> results on an unrelated project, PEBKAC is a
> distinct possibility here, and this part may be a red herring. I'll need
> to look into it.

And just so there's no confusion, this is purely about my comments. The
problem with reproducible builds Bernhard presents is still a problem.

One of the key questions is whether we end up launching multiple
sphinx-build processes ourselves (but maybe some other way than I
speculated), making this a duplicate of [1], or whether plain single
sphinx-build -j<N> on its own *also* has parallel build issues. And if
the latter, can they be caused by our extensions incorrectly indicating
parallel_read_safe = True or parallel_write_safe = True in their setup()
functions, or is it something inherently in Sphinx.

BR,
Jani.


[1] https://github.com/sphinx-doc/sphinx/issues/2946

-- 
Jani Nikula, Intel

Re: [PATCH] docs: Build kernel docs deterministically

[Vegard Nossum]

On 05/09/2024 13:35, bernhard+linux-doc@lsmod.de wrote:
> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
> 
> Because we want reproducible builds
> and https://github.com/sphinx-doc/sphinx/issues/6714
> did not receive any love from Sphinx devs in five years,
> let's disable parallel doc builds until that Sphinx issue is fixed.
> 
> This patch was done while working on reproducible builds for openSUSE,
> sponsored by the NLnet NGI0 fund.
> 
> Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
> ---
>   Documentation/sphinx/parallel-wrapper.sh | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphinx/parallel-wrapper.sh
> index e54c44ce117d..cb93626bd86e 100644
> --- a/Documentation/sphinx/parallel-wrapper.sh
> +++ b/Documentation/sphinx/parallel-wrapper.sh
> @@ -10,6 +10,9 @@ sphinx="$1"
>   shift || true
>   
>   parallel="$PARALLELISM"
> +# Because of issues in Sphinx(https://github.com/sphinx-doc/sphinx/issues/6714)
> +# we disable parallel doc generation to get deterministic build results
> +parallel=1
>   if [ -z "$parallel" ] ; then
>   	# If no parallelism is specified at the top-level make, then
>   	# fall back to the expected "-jauto" mode that the "htmldocs"

If you need reproducibility and the solution/workaround is to limit it
to 1 concurrent sphinx-build invocation, why can't you just run
'make -j1 htmldocs ...' explicitly? (or whichever doc targets you need.)
This should also have the benefit of working regardless of the exact
kernel version.


Vegard

Re: [PATCH] docs: Build kernel docs deterministically

[Bernhard M. Wiedemann]

On 05/09/2024 20.01, Jonathan Corbet wrote:
> [I wouldn't have minded being CC'd on this conversation...]

Will do next time.

Looking at one of the diffs I got:
https://rb.zq1.de/compare.factory-20240904/diffs/kernel-source-compare.out
whatisRCU.html seems to have a dozen missing lines in one case.
Possibly, because it depends on which other files have already been 
processed at that moment.

Ciao
Bernhard M.

Re: [PATCH] docs: Build kernel docs deterministically

[Bernhard M. Wiedemann]

On 06/09/2024 11.11, Vegard Nossum wrote:
> On 05/09/2024 13:35, bernhard+linux-doc@lsmod.de wrote:
>> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>

> If you need reproducibility and the solution/workaround is to limit it
> to 1 concurrent sphinx-build invocation, why can't you just run
> 'make -j1 htmldocs ...' explicitly?

I had tried that and it did not help.

however, an
export PARALLELISM=1
did help to make results reproducible.
This is converted in parallel-wrapper.sh into a sphinx -j1 $ARGS
call instead of sphinx -jauto.
And that points towards sphinx-internal parallelism as the issue here, 
rather than parallelism from kernel/make.

And that means https://github.com/sphinx-doc/sphinx/issues/6714
for kernel-docs, is not exactly the same issue as
https://github.com/sphinx-doc/sphinx/issues/2946


While I could add that workaround into our openSUSE kernel-docs.spec 
file, there are still Debian, Archlinux and others that desire 
reproducible builds and those would need to discover and apply that 
workaround as well. So would be nicer to have it reproducible by 
default, possibly with an option to sacrifice correct results for 
processing-speed.

Ciao
Bernhard M.

Re: [PATCH] docs: Build kernel docs deterministically

[Akira Yokosawa]

Hi,

On Fri, 6 Sep 2024 15:56:10 +0200, Bernhard M. Wiedemann wrote:
[...]
> On 06/09/2024 11.11, Vegard Nossum wrote:
> > On 05/09/2024 13:35, bernhard+linux-doc@lsmod.de wrote:
> >> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>
> > If you need reproducibility and the solution/workaround is to limit it
> > to 1 concurrent sphinx-build invocation, why can't you just run
> > 'make -j1 htmldocs ...' explicitly?
> > 
> I had tried that and it did not help.
> 

I think 'make SPHINXOPTS="-q -j1" htmldocs' should work for you.

This is more of a happens-to-work kind of hack, which assumes that
sphinx-build picks up a final -j option in the command line if there
are several.

Actual command line invoked from parallel-wrapper.sh will look something
like (wrapping by me):

    sphinx-build -jauto -b html -c /<abspath>/linux/Documentation \
	-d /<abspath>/linux/Documentation/output/.doctrees \
	-D version=6.11.0-rc6 -D release= -D kerneldoc_srctree=. \
	-D kerneldoc_bin=./scripts/kernel-doc -q -j1 \
	/<abspath>/linux/Documentation \
	/<abspath>/linux/Documentation/output

You see, -jauto (from $PARALLELISM) will be superseded by the -j1
(from $SPHINXOPTS).

Hope this helps.

        Akira

[PATCH] docs/zh_TW+zh_CN: Make rst references unique

[bernhard+linux-doc]

From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>

We found that sphinx parallel processing would randomly pick
one or the other
https://github.com/sphinx-doc/sphinx/issues/6714#issuecomment-2362212754

Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
---
 Documentation/translations/zh_CN/dev-tools/gcov.rst | 8 ++++----
 Documentation/translations/zh_TW/dev-tools/gcov.rst | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/Documentation/translations/zh_CN/dev-tools/gcov.rst b/Documentation/translations/zh_CN/dev-tools/gcov.rst
index 3158c9da1318..ea8f94852f41 100644
--- a/Documentation/translations/zh_CN/dev-tools/gcov.rst
+++ b/Documentation/translations/zh_CN/dev-tools/gcov.rst
@@ -120,7 +120,7 @@ gcov的内核分析插桩支持内核的编译和运行是在同一台机器上
 如果内核编译和运行是不同的机器，那么需要额外的准备工作，这取决于gcov工具
 是在哪里使用的:
 
-.. _gcov-test_zh:
+.. _gcov-test_zh_CN:
 
 a) 若gcov运行在测试机上
 
@@ -140,7 +140,7 @@ a) 若gcov运行在测试机上
     如果文件是软链接，需要替换成真正的目录文件（这是由make的当前工作
     目录变量CURDIR引起的）。
 
-.. _gcov-build_zh:
+.. _gcov-build_zh_CN:
 
 b) 若gcov运行在编译机上
 
@@ -205,7 +205,7 @@ kconfig会根据编译工具链的检查自动选择合适的gcov格式。
 --------------------------
 
 用于在编译机上收集覆盖率元文件的示例脚本
-（见 :ref:`编译机和测试机分离 a. <gcov-test_zh>` ）
+（见 :ref:`编译机和测试机分离 a. <gcov-test_zh_CN>` ）
 
 .. code-block:: sh
 
@@ -238,7 +238,7 @@ kconfig会根据编译工具链的检查自动选择合适的gcov格式。
 -------------------------
 
 用于在测试机上收集覆盖率数据文件的示例脚本
-（见 :ref:`编译机和测试机分离 b. <gcov-build_zh>` ）
+（见 :ref:`编译机和测试机分离 b. <gcov-build_zh_CN>` ）
 
 .. code-block:: sh
 
diff --git a/Documentation/translations/zh_TW/dev-tools/gcov.rst b/Documentation/translations/zh_TW/dev-tools/gcov.rst
index ce1c9a97de16..39ac3fff44cd 100644
--- a/Documentation/translations/zh_TW/dev-tools/gcov.rst
+++ b/Documentation/translations/zh_TW/dev-tools/gcov.rst
@@ -120,7 +120,7 @@ gcov的內核分析插樁支持內核的編譯和運行是在同一臺機器上
 如果內核編譯和運行是不同的機器，那麼需要額外的準備工作，這取決於gcov工具
 是在哪裏使用的:
 
-.. _gcov-test_zh:
+.. _gcov-test_zh_TW:
 
 a) 若gcov運行在測試機上
 
@@ -140,7 +140,7 @@ a) 若gcov運行在測試機上
     如果文件是軟鏈接，需要替換成真正的目錄文件（這是由make的當前工作
     目錄變量CURDIR引起的）。
 
-.. _gcov-build_zh:
+.. _gcov-build_zh_TW:
 
 b) 若gcov運行在編譯機上
 
@@ -205,7 +205,7 @@ kconfig會根據編譯工具鏈的檢查自動選擇合適的gcov格式。
 --------------------------
 
 用於在編譯機上收集覆蓋率元文件的示例腳本
-（見 :ref:`編譯機和測試機分離 a. <gcov-test_zh>` ）
+（見 :ref:`編譯機和測試機分離 a. <gcov-test_zh_TW>` ）
 
 .. code-block:: sh
 
@@ -238,7 +238,7 @@ kconfig會根據編譯工具鏈的檢查自動選擇合適的gcov格式。
 -------------------------
 
 用於在測試機上收集覆蓋率數據文件的示例腳本
-（見 :ref:`編譯機和測試機分離 b. <gcov-build_zh>` ）
+（見 :ref:`編譯機和測試機分離 b. <gcov-build_zh_TW>` ）
 
 .. code-block:: sh
 
-- 
2.35.3

Re: [PATCH] docs/zh_TW+zh_CN: Make rst references unique

[Yanteng Si]

在 2024/9/20 15:01, bernhard+linux-doc@lsmod.de 写道:
> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>
> We found that sphinx parallel processing would randomly pick
> one or the other
> https://github.com/sphinx-doc/sphinx/issues/6714#issuecomment-2362212754

It seems to be a problem left over from the initial conversion from

Simplified Chinese to Traditional Chinese.


For Simplified Chinese：

Reviewed-by: Yanteng Si <siyanteng@linux.dev>


Thanks,

Yanteng

>
> Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
> ---
>   Documentation/translations/zh_CN/dev-tools/gcov.rst | 8 ++++----
>   Documentation/translations/zh_TW/dev-tools/gcov.rst | 8 ++++----
>   2 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/translations/zh_CN/dev-tools/gcov.rst b/Documentation/translations/zh_CN/dev-tools/gcov.rst
> index 3158c9da1318..ea8f94852f41 100644
> --- a/Documentation/translations/zh_CN/dev-tools/gcov.rst
> +++ b/Documentation/translations/zh_CN/dev-tools/gcov.rst
> @@ -120,7 +120,7 @@ gcov的内核分析插桩支持内核的编译和运行是在同一台机器上
>   如果内核编译和运行是不同的机器，那么需要额外的准备工作，这取决于gcov工具
>   是在哪里使用的:
>   
> -.. _gcov-test_zh:
> +.. _gcov-test_zh_CN:
>   
>   a) 若gcov运行在测试机上
>   
> @@ -140,7 +140,7 @@ a) 若gcov运行在测试机上
>       如果文件是软链接，需要替换成真正的目录文件（这是由make的当前工作
>       目录变量CURDIR引起的）。
>   
> -.. _gcov-build_zh:
> +.. _gcov-build_zh_CN:
>   
>   b) 若gcov运行在编译机上
>   
> @@ -205,7 +205,7 @@ kconfig会根据编译工具链的检查自动选择合适的gcov格式。
>   --------------------------
>   
>   用于在编译机上收集覆盖率元文件的示例脚本
> -（见 :ref:`编译机和测试机分离 a. <gcov-test_zh>` ）
> +（见 :ref:`编译机和测试机分离 a. <gcov-test_zh_CN>` ）
>   
>   .. code-block:: sh
>   
> @@ -238,7 +238,7 @@ kconfig会根据编译工具链的检查自动选择合适的gcov格式。
>   -------------------------
>   
>   用于在测试机上收集覆盖率数据文件的示例脚本
> -（见 :ref:`编译机和测试机分离 b. <gcov-build_zh>` ）
> +（见 :ref:`编译机和测试机分离 b. <gcov-build_zh_CN>` ）
>   
>   .. code-block:: sh
>   
> diff --git a/Documentation/translations/zh_TW/dev-tools/gcov.rst b/Documentation/translations/zh_TW/dev-tools/gcov.rst
> index ce1c9a97de16..39ac3fff44cd 100644
> --- a/Documentation/translations/zh_TW/dev-tools/gcov.rst
> +++ b/Documentation/translations/zh_TW/dev-tools/gcov.rst
> @@ -120,7 +120,7 @@ gcov的內核分析插樁支持內核的編譯和運行是在同一臺機器上
>   如果內核編譯和運行是不同的機器，那麼需要額外的準備工作，這取決於gcov工具
>   是在哪裏使用的:
>   
> -.. _gcov-test_zh:
> +.. _gcov-test_zh_TW:
>   
>   a) 若gcov運行在測試機上
>   
> @@ -140,7 +140,7 @@ a) 若gcov運行在測試機上
>       如果文件是軟鏈接，需要替換成真正的目錄文件（這是由make的當前工作
>       目錄變量CURDIR引起的）。
>   
> -.. _gcov-build_zh:
> +.. _gcov-build_zh_TW:
>   
>   b) 若gcov運行在編譯機上
>   
> @@ -205,7 +205,7 @@ kconfig會根據編譯工具鏈的檢查自動選擇合適的gcov格式。
>   --------------------------
>   
>   用於在編譯機上收集覆蓋率元文件的示例腳本
> -（見 :ref:`編譯機和測試機分離 a. <gcov-test_zh>` ）
> +（見 :ref:`編譯機和測試機分離 a. <gcov-test_zh_TW>` ）
>   
>   .. code-block:: sh
>   
> @@ -238,7 +238,7 @@ kconfig會根據編譯工具鏈的檢查自動選擇合適的gcov格式。
>   -------------------------
>   
>   用於在測試機上收集覆蓋率數據文件的示例腳本
> -（見 :ref:`編譯機和測試機分離 b. <gcov-build_zh>` ）
> +（見 :ref:`編譯機和測試機分離 b. <gcov-build_zh_TW>` ）
>   
>   .. code-block:: sh
>   

Re: [PATCH] docs/zh_TW+zh_CN: Make rst references unique

[Jonathan Corbet]

bernhard+linux-doc@lsmod.de writes:

> From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
>
> We found that sphinx parallel processing would randomly pick
> one or the other
> https://github.com/sphinx-doc/sphinx/issues/6714#issuecomment-2362212754
>
> Signed-off-by: Bernhard M. Wiedemann <bwiedemann@suse.de>
> ---
>  Documentation/translations/zh_CN/dev-tools/gcov.rst | 8 ++++----
>  Documentation/translations/zh_TW/dev-tools/gcov.rst | 8 ++++----
>  2 files changed, 8 insertions(+), 8 deletions(-)

Applied, thanks.

jon