Re: Sphinx parallel build errors

[Randy Dunlap <rdunlap@infradead.org>]

On 10/29/19 1:16 AM, Markus Heiser wrote:
> Am 28.10.19 um 20:07 schrieb Kees Cook:
>> On Sun, Oct 27, 2019 at 04:05:07PM -0300, Mauro Carvalho Chehab wrote:
>>> Em Sun, 27 Oct 2019 11:30:22 -0700
>>> Randy Dunlap <rdunlap@infradead.org> escreveu:
>>>
>>>> On 10/27/19 11:17 AM, Jonathan Corbet wrote:
>>>>> On Sun, 27 Oct 2019 11:11:57 -0700
>>>>> Randy Dunlap <rdunlap@infradead.org> wrote:
>>>>>   
>>>>>>> Yes, I see oom-kills when building docs for 5.4-rc2.
>>>>>>> But only after I ran 'gimp *.jpg' for 22 photos.
>>>>>>>
>>>>>>> I can see anywhere from 1 to 4 sphinx-build processes running.
>>>>>>
>>>>>> I continue to have oom-kills when building docs.
>>>
>>> My 2 cents here.
>>>
>>> I would keep the default to use more CPUs, as nowadays I suspect that
>>> most devs have a way more than 8GB installed. The worse machine I use
>>> here to build Kernel is a Gen-3 i5 that has has 16GB (and it is a
>>> personal laptop I bought 5 years ago).
>>>
>>> Yet, I agree that doc build should be saner on low-cap machines.
>>>
> 
> In the past I run sphinx builds on a small 4GB VM (2cpu) without any problems.
> (nowadays) 16 CPU, 16 GB PC with sphinx-build the max load of my whole PC is
> about 5GB (*more CPUs need more mem*).
> 
> @Randy you said the build machine hase 8GB .. and how much CPUs?

4.

>>> Ideally, the fix should be, IMHO, at sphinx-build: -jauto should
>>> detect OOM possible conditions and reduce the number of used CPUs
>>> it the max available memory is too low.
>>
>> How can we determine that dynamically, though? e.g. how do we determine
>> how much is needed per thread in a way that will move with the future
>> needs of Sphinx, etc?
> 
> d'accord
> 
>>> As there's now a script at Kernel, maybe the script could be modified
>>> to detect if the max available memory is too low, passing -j1 if,
>>> let's say, the machine has less than (let's say) 16 GB RAM.
>>
>> Randy, are you able to just use -j2 or similar with your builds to avoid
>> this?

No, sometimes even -j1 fails with oom-killer.

> 
> Sphinx-doc is pure Python, it runs a lot of regexpr and is moving string,
> nothing special .. maybe we have a mem leak in one of our extensions?
> 
> @Kess: why did you mentioning MAINTAINERS parsing .. did you observe anything
> strange?
> 
> Maybe one of the tools used by sphinx is buggy (latex, dot, image-magic) .. needs investigation.
> 
> @Randy: my first questions are:
> 
> - cpu / mem?
> - are OOM-kills when building target htmldocs, pdfdocs or ..?

htmldocs

> - which py & sphinx version are used? Reference installation see 'virtualenv'
>   in [1]

Python 2.7.14
sphinx-build 1.7.6

> -- Markus --
> 
> 
> [1] https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html#sphinx-install

thanks.
-- 
~Randy