Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Em Mon, 01 Sep 2025 13:53:41 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > It should be noticed that kernel-doc doesn't run in parallel. Python
> > still suffers for a global big lock (called GIL). My attempts to run in
> > parallel actually made kernel-doc slower, but this is changing: the
> > next Python version is planning to get rid of GIL. So, maybe within
> > a year we can re-add the patches to run it in parallel.  
> 
> I certainly wouldn't want to discourage work in this area, but I do
> wonder if it would be worth the trouble; kernel-doc is nowhere near
> being the bottleneck in this whole process. 

I know kernel-doc is not a bottleneck. When I wrote it, I wrote to
run in parallel. The original patches supported even both 
multiprocess and multithread, and had command line parameters to
adjust, but I got disappointed with results, so I dropped it.
Re-adding shoudn't be hard, as I tried to ensure that this would
work since the beginning.

This is, btw, one of the main reasons why I created a separate
class to handle files and used an interactor for file names: the
entire parser logic is confined to one instance per file and
multiple instances can run in parallel. This way concurrent.futures
can distribute files to different processes and threads.

> Now, if you could
> multi-thread the Sphinx HTML builder...

Agreed. I suspect that, if Sphinx can run without GIL, in the future,
we'll have huge performance gains, specially on machines with multiple
CPU cores.

Thanks,
Mauro