Re: [PATCH] docs: Build kernel docs deterministically

[Jani Nikula <jani.nikula@linux.intel.com>]

On Thu, 05 Sep 2024, Jani Nikula <jani.nikula@linux.intel.com> wrote:
> On Thu, 05 Sep 2024, Jonathan Corbet <corbet@lwn.net> wrote:
>> Jani Nikula <jani.nikula@linux.intel.com> writes:
>>
>>>> However my patch for sphinx -j1 did give good test results, too.
>>>> Maybe in your case that would result in 8 sphinx calls with 1 thread 
>>>> each, which would be more appropriate for your machine.
>>>
>>> The right thing to do is to have one sphinx-build process and pass -j<N>
>>> to that.
>>
>> [I wouldn't have minded being CC'd on this conversation...]
>
> Yes, sorry about that.
>
>> I, too, have never seen the behavior Jani reports.
>>
>> I, too, would like to get rid of as much of the makefile hackery as
>> possible, but it all did end up there for a reason.
>>
>> The business around parallelism was intended to make sphinx play well
>> with other targets being built in the same make invocation.  If you do a
>> "make -j8 this that theother htmldocs" and there are five processes
>> working on this, that, and theother, then sphinx should not create more
>> than three.  See 51e46c7a4007 for more.
>>
>> In Jani's case, it sounds like the job-slot reservation isn't working
>> right somehow?
>
> Looking at the -j<N> results on an unrelated project, PEBKAC is a
> distinct possibility here, and this part may be a red herring. I'll need
> to look into it.

And just so there's no confusion, this is purely about my comments. The
problem with reproducible builds Bernhard presents is still a problem.

One of the key questions is whether we end up launching multiple
sphinx-build processes ourselves (but maybe some other way than I
speculated), making this a duplicate of [1], or whether plain single
sphinx-build -j<N> on its own *also* has parallel build issues. And if
the latter, can they be caused by our extensions incorrectly indicating
parallel_read_safe = True or parallel_write_safe = True in their setup()
functions, or is it something inherently in Sphinx.

BR,
Jani.


[1] https://github.com/sphinx-doc/sphinx/issues/2946

-- 
Jani Nikula, Intel