Re: [PATCH 0/4] Improve Sphinx kerneldoc extension

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Em Mon, 21 Apr 2025 11:42:09 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Hi Jon,
> >
> > As promised, this series improves the Sphinx kerneldoc extension making it using the
> > kernel-doc.py classes directly if available.
> >
> > The script still supports excecuting kernel-doc via shell, and, in verbose mode, it will
> > show the command line arguments to run kernel-doc manually, even when the Python
> > classes are used. The idea is that the command line verbose will help to eventually
> > debug issues if one needs to run kernel-doc.py manually. 
> >
> > On other words, after this series, if one does:
> >
> > 	make htmldocs KERNELDOC=scripts/kernel-doc.py
> >
> > Or, simply (as internally KERNELDOC is set to scripts/kernel-doc.py):
> >
> > 	make htmldocs
> >
> > It will use the Python classes instead of running a subprocess.
> >
> > If one uses, instead:
> >
> > 	make htmldocs KERNELDOC=scripts/kernel-doc
> > or:
> > 	make htmldocs KERNELDOC=scripts/kernel-doc.pl
> >
> > As the file extension doesn't end with .py, it will excecute the Python or the Perl
> > version of kernel-doc via a subprocess.
> >
> > On this version, I opted to re-create the Python objects for every single kernel-doc
> > line, so no caches from past runs are used. I'm working on a version that will cache
> > results, but it is currently causing some regressions. So, let's do the changes
> > step-by-step, applying first this improvement patch series.
> >
> > PS.: the first patches on this series are addressing some some bugs and one
> > improvement that I noticed while debugging the patch changing kerneldoc
> > Sphinx extension.
> >
> > Mauro Carvalho Chehab (4):
> >   scripts/lib/kdoc/kdoc_files.py: don't try to join None
> >   scripts/lib/kdoc/kdoc_parser.py: move states to a separate class
> >   scripts:kdoc_files.py: use glob for export_file seek
> >   docs: sphinx: kerneldoc: Use python class if available
> >
> >  Documentation/sphinx/kerneldoc.py | 138 ++++++++++++++++++++++++++++--
> >  scripts/lib/kdoc/kdoc_files.py    |  11 ++-
> >  scripts/lib/kdoc/kdoc_parser.py   | 119 ++++++++++++++------------
> >  3 files changed, 200 insertions(+), 68 deletions(-)  
> 
> I've applied the series.
> 
> I do note that the default "make htmldocs" build time is reliably slower
> than with KERNELDOC=scripts/kernel-doc, I'd be curious to understand
> why.
> 
>         External kdoc:  170s
>         w/classes:	186s

I noticed that. My guess is that this has to do with Python's big lock
(GIL). When it uses processes, there won't be any internal locks, as
kernel-doc will run independently. There is an effort for Python to get
rid of GIL, but it seems GIL-less threads won't be default on version 3.14.

-

That's said, please notice that I intend to work on some patches that
will optimize that. Right now, on some places, the same file is processed
multiple times. By using the classes, we can cache the processing results
the first time and then re-use them every time a kernel-doc tag for the
same file is found within Documentation.

This is easier said than done. I did one attempt to do that, but it ended
causing troubles. So, I'm working on a new version of it.

Regards,
Mauro