Re: [PATCH v3 00/33] Implement kernel-doc in Python

[Jani Nikula <jani.nikula@linux.intel.com>]

On Tue, 08 Apr 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Hi Jon,
>
> This changeset contains the kernel-doc.py script to replace the verable
> kernel-doc originally written in Perl. It replaces the first version and the
> second series I sent on the top of it.

Yay! Thanks for doing this. I believe this will make contributing to
kernel-doc more accessible in the long run.

> I tried to stay as close as possible of the original Perl implementation
> on the first patch introducing kernel-doc.py, as it helps to double check
> if each function was  properly translated to Python.  This have been 
> helpful debugging troubles that happened during the conversion.
>
> I worked hard to make it bug-compatible with the original one. Still, its
> output has a couple of differences from the original one:
>
> - The tab expansion works better with the Python script. With that, some
>   outputs that contain tabs at kernel-doc markups are now different;
>
> - The new script  works better stripping blank lines. So, there are a couple
>   of empty new lines that are now stripped with this version;
>
> - There is a buggy logic at kernel-doc to strip empty description and
>   return sections. I was not able to replicate the exact behavior. So, I ended
>   adding an extra logic to strip empty sections with a different algorithm.
>
> Yet, on my tests, the results are compatible with the venerable script
> output for all .. kernel-doc tags found in Documentation/. I double-checked
> this by adding support to output the kernel-doc commands when V=1, and
> then I ran a diff between kernel-doc.pl and kernel-doc.py for the same
> command lines.
>
> The only patch that doesn't belong to this series is a patch dropping
> kernel-doc.pl. I opted to keep it for now, as it can help to better
> test the new tools.
>
> With such changes, if one wants to build docs with the old script,
> all it is needed is to use KERNELDOC parameter, e.g.:
>
> 	$ make KERNELDOC=scripts/kernel-doc.pl htmldocs

I guess that's good for double checking that the python version
reproduces the output of the old version, warts and all. And it could be
used standalone for comparing the output for .[ch] files directly
instead of going through Sphinx.

But once we're reasonably sure the new one works fine, I think the
natural follow-up will be to import the kernel-doc python module from
the kernel-doc Sphinx extension instead of running it with
subprocess.Popen(). It'll bypass an absolutely insane amount of forks,
python interpreter launches and module imports.

It'll also open the door for passing the results in python native
structures instead of text, also making it possible to cache parse
results instead of parsing the source files for every kernel-doc
directive in rst.

Another idea regarding code organization, again for future. Maybe we
should have a scripts/python/ directory structure, so we can point
python path there, and be able to import stuff from there? And
reasonably share code between modules. And have linters handle it
recursively, etc, etc.

Anyway, I applaud the work, and I regret that I don't have time to
review it in detail. Regardless, I think the matching output is the most
important part.


BR,
Jani.

> ---
>
> v3:
> - rebased on the top of v6.15-rc1;
> - Removed patches that weren't touching kernel-doc and its Sphinx extension;
> - The "Re" class was renamed to "KernRe"
> - It contains one patch from Sean with an additional hunk for the
>   python version.
>
> Mauro Carvalho Chehab (32):
>   scripts/kernel-doc: rename it to scripts/kernel-doc.pl
>   scripts/kernel-doc: add a symlink to the Perl version of kernel-doc
>   scripts/kernel-doc.py: add a Python parser
>   scripts/kernel-doc.py: output warnings the same way as kerneldoc
>   scripts/kernel-doc.py: better handle empty sections
>   scripts/kernel-doc.py: properly handle struct_group macros
>   scripts/kernel-doc.py: move regex methods to a separate file
>   scripts/kernel-doc.py: move KernelDoc class to a separate file
>   scripts/kernel-doc.py: move KernelFiles class to a separate file
>   scripts/kernel-doc.py: move output classes to a separate file
>   scripts/kernel-doc.py: convert message output to an interactor
>   scripts/kernel-doc.py: move file lists to the parser function
>   scripts/kernel-doc.py: implement support for -no-doc-sections
>   scripts/kernel-doc.py: fix line number output
>   scripts/kernel-doc.py: fix handling of doc output check
>   scripts/kernel-doc.py: properly handle out_section for ReST
>   scripts/kernel-doc.py: postpone warnings to the output plugin
>   docs: add a .pylintrc file with sys path for docs scripts
>   docs: sphinx: kerneldoc: verbose kernel-doc command if V=1
>   docs: sphinx: kerneldoc: ignore "\" characters from options
>   docs: sphinx: kerneldoc: use kernel-doc.py script
>   scripts/kernel-doc.py: Set an output format for --none
>   scripts/kernel-doc.py: adjust some coding style issues
>   scripts/lib/kdoc/kdoc_parser.py: fix Python compat with < v3.13
>   scripts/kernel-doc.py: move modulename to man class
>   scripts/kernel-doc.py: properly handle KBUILD_BUILD_TIMESTAMP
>   scripts/lib/kdoc/kdoc_parser.py: remove a python 3.9 dependency
>   scripts/kernel-doc.py: Properly handle Werror and exit codes
>   scripts/kernel-doc: switch to use kernel-doc.py
>   scripts/lib/kdoc/kdoc_files.py: allow filtering output per fname
>   scripts/kernel_doc.py: better handle exported symbols
>   scripts/kernel-doc.py: Rename the kernel doc Re class to KernRe
>
> Sean Anderson (1):
>   scripts: kernel-doc: fix parsing function-like typedefs (again)
>
>  .pylintrc                         |    2 +
>  Documentation/Makefile            |    2 +-
>  Documentation/conf.py             |    2 +-
>  Documentation/sphinx/kerneldoc.py |   46 +
>  scripts/kernel-doc                | 2440 +----------------------------
>  scripts/kernel-doc.pl             | 2439 ++++++++++++++++++++++++++++
>  scripts/kernel-doc.py             |  315 ++++
>  scripts/lib/kdoc/kdoc_files.py    |  282 ++++
>  scripts/lib/kdoc/kdoc_output.py   |  793 ++++++++++
>  scripts/lib/kdoc/kdoc_parser.py   | 1715 ++++++++++++++++++++
>  scripts/lib/kdoc/kdoc_re.py       |  273 ++++
>  11 files changed, 5868 insertions(+), 2441 deletions(-)
>  create mode 100644 .pylintrc
>  mode change 100755 => 120000 scripts/kernel-doc
>  create mode 100755 scripts/kernel-doc.pl
>  create mode 100755 scripts/kernel-doc.py
>  create mode 100644 scripts/lib/kdoc/kdoc_files.py
>  create mode 100755 scripts/lib/kdoc/kdoc_output.py
>  create mode 100755 scripts/lib/kdoc/kdoc_parser.py
>  create mode 100755 scripts/lib/kdoc/kdoc_re.py

-- 
Jani Nikula, Intel