Re: [PATCH v3 00/33] Implement kernel-doc in Python

Mauro Carvalho Chehab
2025-04-09T11:44:47Z

Em Wed, 09 Apr 2025 13:16:06 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> On Tue, 08 Apr 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Hi Jon,
> >
> > This changeset contains the kernel-doc.py script to replace the verable
> > kernel-doc originally written in Perl. It replaces the first version and the
> > second series I sent on the top of it.
>
> Yay! Thanks for doing this. I believe this will make contributing to
> kernel-doc more accessible in the long run.
>
> > I tried to stay as close as possible of the original Perl implementation
> > on the first patch introducing kernel-doc.py, as it helps to double check
> > if each function was properly translated to Python. This have been
> > helpful debugging troubles that happened during the conversion.
> >
> > I worked hard to make it bug-compatible with the original one. Still, its
> > output has a couple of differences from the original one:
> >
> > - The tab expansion works better with the Python script. With that, some
> > outputs that contain tabs at kernel-doc markups are now different;
> >
> > - The new script works better stripping blank lines. So, there are a couple
> > of empty new lines that are now stripped with this version;
> >
> > - There is a buggy logic at kernel-doc to strip empty description and
> > return sections. I was not able to replicate the exact behavior. So, I ended
> > adding an extra logic to strip empty sections with a different algorithm.
> >
> > Yet, on my tests, the results are compatible with the venerable script
> > output for all .. kernel-doc tags found in Documentation/. I double-checked
> > this by adding support to output the kernel-doc commands when V=1, and
> > then I ran a diff between kernel-doc.pl and kernel-doc.py for the same
> > command lines.
> >
> > The only patch that doesn't belong to this series is a patch dropping
> > kernel-doc.pl. I opted to keep it for now, as it can help to better
> > test the new tools.
> >
> > With such changes, if one wants to build docs with the old script,
> > all it is needed is to use KERNELDOC parameter, e.g.:
> >
> > $ make KERNELDOC=scripts/kernel-doc.pl htmldocs
>
> I guess that's good for double checking that the python version
> reproduces the output of the old version, warts and all. And it could be
> used standalone for comparing the output for .[ch] files directly
> instead of going through Sphinx.
>
> But once we're reasonably sure the new one works fine, I think the
> natural follow-up will be to import the kernel-doc python module from
> the kernel-doc Sphinx extension instead of running it with
> subprocess.Popen(). It'll bypass an absolutely insane amount of forks,
> python interpreter launches and module imports.
>
> It'll also open the door for passing the results in python native
> structures instead of text, also making it possible to cache parse
> results instead of parsing the source files for every kernel-doc
> directive in rst.

Yes, this is on my plan. I have already a patch series for that,
but it still requires some care to ensure that the results will be
identical.

> Another idea regarding code organization, again for future. Maybe we
> should have a scripts/python/ directory structure, so we can point
> python path there, and be able to import stuff from there? And
> reasonably share code between modules. And have linters handle it
> recursively, etc, etc.

Sounds like a plan. I did some code reorg already, but surely there
are spaces for improvements.

> Anyway, I applaud the work, and I regret that I don't have time to
> review it in detail. Regardless, I think the matching output is the most
> important part.

I did several tests here to check the output, making it similar to the
output from the Perl version.

>
>
> BR,
> Jani.
>
> > ---
> >
> > v3:
> > - rebased on the top of v6.15-rc1;
> > - Removed patches that weren't touching kernel-doc and its Sphinx extension;
> > - The "Re" class was renamed to "KernRe"
> > - It contains one patch from Sean with an additional hunk for the
> > python version.
> >
> > Mauro Carvalho Chehab (32):
> > scripts/kernel-doc: rename it to scripts/kernel-doc.pl
> > scripts/kernel-doc: add a symlink to the Perl version of kernel-doc
> > scripts/kernel-doc.py: add a Python parser
> > scripts/kernel-doc.py: output warnings the same way as kerneldoc
> > scripts/kernel-doc.py: better handle empty sections
> > scripts/kernel-doc.py: properly handle struct_group macros
> > scripts/kernel-doc.py: move regex methods to a separate file
> > scripts/kernel-doc.py: move KernelDoc class to a separate file
> > scripts/kernel-doc.py: move KernelFiles class to a separate file
> > scripts/kernel-doc.py: move output classes to a separate file
> > scripts/kernel-doc.py: convert message output to an interactor
> > scripts/kernel-doc.py: move file lists to the parser function
> > scripts/kernel-doc.py: implement support for -no-doc-sections
> > scripts/kernel-doc.py: fix line number output
> > scripts/kernel-doc.py: fix handling of doc output check
> > scripts/kernel-doc.py: properly handle out_section for ReST
> > scripts/kernel-doc.py: postpone warnings to the output plugin
> > docs: add a .pylintrc file with sys path for docs scripts
> > docs: sphinx: kerneldoc: verbose kernel-doc command if V=1
> > docs: sphinx: kerneldoc: ignore "\" characters from options
> > docs: sphinx: kerneldoc: use kernel-doc.py script
> > scripts/kernel-doc.py: Set an output format for --none
> > scripts/kernel-doc.py: adjust some coding style issues
> > scripts/lib/kdoc/kdoc_parser.py: fix Python compat with < v3.13
> > scripts/kernel-doc.py: move modulename to man class
> > scripts/kernel-doc.py: properly handle KBUILD_BUILD_TIMESTAMP
> > scripts/lib/kdoc/kdoc_parser.py: remove a python 3.9 dependency
> > scripts/kernel-doc.py: Properly handle Werror and exit codes
> > scripts/kernel-doc: switch to use kernel-doc.py
> > scripts/lib/kdoc/kdoc_files.py: allow filtering output per fname
> > scripts/kernel_doc.py: better handle exported symbols
> > scripts/kernel-doc.py: Rename the kernel doc Re class to KernRe
> >
> > Sean Anderson (1):
> > scripts: kernel-doc: fix parsing function-like typedefs (again)
> >
> > .pylintrc | 2 +
> > Documentation/Makefile | 2 +-
> > Documentation/conf.py | 2 +-
> > Documentation/sphinx/kerneldoc.py | 46 +
> > scripts/kernel-doc | 2440 +----------------------------
> > scripts/kernel-doc.pl | 2439 ++++++++++++++++++++++++++++
> > scripts/kernel-doc.py | 315 ++++
> > scripts/lib/kdoc/kdoc_files.py | 282 ++++
> > scripts/lib/kdoc/kdoc_output.py | 793 ++++++++++
> > scripts/lib/kdoc/kdoc_parser.py | 1715 ++++++++++++++++++++
> > scripts/lib/kdoc/kdoc_re.py | 273 ++++
> > 11 files changed, 5868 insertions(+), 2441 deletions(-)
> > create mode 100644 .pylintrc
> > mode change 100755 => 120000 scripts/kernel-doc
> > create mode 100755 scripts/kernel-doc.pl
> > create mode 100755 scripts/kernel-doc.py
> > create mode 100644 scripts/lib/kdoc/kdoc_files.py
> > create mode 100755 scripts/lib/kdoc/kdoc_output.py
> > create mode 100755 scripts/lib/kdoc/kdoc_parser.py
> > create mode 100755 scripts/lib/kdoc/kdoc_re.py
>