Re: [PATCH for-10.2 0/8] docs: Update our kernel-doc script to the kernel's new Python one

[Paolo Bonzini <pbonzini@redhat.com>]

On 8/14/25 19:13, Peter Maydell wrote:
> Earlier this year, the Linux kernel's kernel-doc script was rewritten
> from the old Perl version into a shiny and hopefully more maintainable
> Python version. This commit series updates our copy of this script
> to the latest kernel version. I have tested it by comparing the
> generated HTML documentation and checking that there are no
> unexpected changes.
> 
> Luckily we are carrying very few local modifications to the Perl
> script, so this is fairly straightforward. The structure of the
> patchset is:
>   * a minor update to the kerneldoc.py Sphinx extension so it
>     will work with both old and new kernel-doc script output
>   * a fix to a doc comment markup error that I noticed while comparing
>     the HTML output from the two versions of the script
>   * import the new Python script, unmodified from the kernel's version
>     (conveniently the kernel calls it kernel-doc.py, so it doesn't
>     clash with the existing script)
>   * make the changes to that library code that correspond to the
>     two local QEMU-specific changes we carry
>   * tell sphinx to use the Python version
>   * delete the Perl script (I have put a diff of our local mods
>     to the Perl script in the commit message of this commit, for
>     posterity)
> 
> The diffstat looks big, but almost all of it is "import the
> kernel's new script that we trust and don't need to review in
> detail" and "delete the old script".
> 
> My immediate motivation for doing this update is that I noticed
> that the submitter of https://gitlab.com/qemu-project/qemu/-/issues/3077
> is using a Perl that complains about a construct in the perl script,
> which prompted me to check if the kernel folks had already fixed
> it, which it turned out that they had, by rewriting the whole thing :-)
> More generally, if we don't do this update, then we're effectively
> going to drift down the same path we did with checkpatch.pl, where
> we have our own version that diverges from the kernel's version
> and we have to maintain it ourselves.

Yep - for checkpatch.pl that makes sense, since we have more differences 
in what we test and we have backported the most pressing parser fixes, 
but kerneldoc has no reason to diverge.

Thanks for doing this!  For the whole series...

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo

> We should also update the Sphinx plugin itself (i.e.
> docs/sphinx/kerneldoc.py), but because I did not need to do
> that to update the main kernel-doc script, I have left that as
> a separate todo item.
> 
> Testing
> -------
> 
> I looked at the HTML output of the old kernel-doc script versus the
> new one, using the following diff command which mechanically excludes
> a couple of "same minor change" everywhere diffs, and eyeballing the
> resulting ~150 lines of diff.
> 
> diff -w  -I '^<div class="kernelindent docutils container">$' -I '^</div>$' -I '^<p><strong>Definition</strong>' -r -u -x searchindex.js build/x86/docs-old-kerneldoc/manual build/x86/docs/manual
> 
> The HTML changes are:
> 
> (1) some paras now have ID tags, eg:
> -<p><strong>Functions operating on arrays of bits</strong></p>
> +<p id="functions-operating-on-arrays-of-bits"><strong>Functions operating on arrays of bits</strong></p>
> 
> (2) Some extra named <div>s, eg:
> +<div class="kernelindent docutils container">
>   <p><strong>Parameters</strong></p>
>   <dl class="simple">
>   <dt><code class="docutils literal notranslate"><span class="pre">long</span> <span class="pre">nr</span></code></dt><dd><p>the bit to set</p>
> @@ -144,12 +145,14 @@
>   <dt><code class="docutils literal notranslate"><span class="pre">unsigned</span> <span class="pre">long</span> <span class="pre">*addr</span></code></dt><dd><p>the address to start counting from</p>
>   </dd>
>   </dl>
> +</div>
> 
> (3) The new version correctly parses the multi-line Return: block for
> the memory_translate_iotlb() doc comment. You can see that the
> old HTML here had dt/dd markup, and it mis-renders in the HTML at
> https://www.qemu.org/docs/master/devel/memory.html#c.memory_translate_iotlb
> 
>   <p><strong>Return</strong></p>
> -<dl class="simple">
> -<dt>On success, return the MemoryRegion containing the <strong>iotlb</strong> translated</dt><dd><p>addr.  The MemoryRegion must not be
>   accessed after rcu_read_unlock.
> +<p>On success, return the MemoryRegion containing the <strong>iotlb</strong> translated
> +addr.  The MemoryRegion must not be accessed after rcu_read_unlock.
>   On failure, return NULL, setting <strong>errp</strong> with error.</p>
> -</dd>
> -</dl>
> +</div>
> 
> "Definition" sections now get output with a trailing colon:
> 
> -<p><strong>Definition</strong></p>
> +<div class="kernelindent docutils container">
> +<p><strong>Definition</strong>:</p>
> 
> This seems like it might be a bug in kernel-doc since the Parameters,
> Return, etc sections don't get the trailing colon. I don't think it's
> important enough to worry about.
> 
> thanks
> -- PMM
> 
> Peter Maydell (8):
>    docs/sphinx/kerneldoc.py: Handle new LINENO syntax
>    tests/qtest/libqtest.h: Remove stray space from doc comment
>    scripts: Import Python kerneldoc from Linux kernel
>    scripts/kernel-doc: strip QEMU_ from function definitions
>    scripts/kernel-doc: tweak for QEMU coding standards
>    scripts/kerneldoc: Switch to the Python kernel-doc script
>    scripts/kernel-doc: Delete the old Perl kernel-doc script
>    MAINTAINERS: Put kernel-doc under the "docs build machinery" section
> 
>   MAINTAINERS                     |    2 +
>   docs/conf.py                    |    4 +-
>   docs/sphinx/kerneldoc.py        |    7 +-
>   tests/qtest/libqtest.h          |    2 +-
>   .editorconfig                   |    2 +-
>   scripts/kernel-doc              | 2442 -------------------------------
>   scripts/kernel-doc.py           |  325 ++++
>   scripts/lib/kdoc/kdoc_files.py  |  291 ++++
>   scripts/lib/kdoc/kdoc_item.py   |   42 +
>   scripts/lib/kdoc/kdoc_output.py |  749 ++++++++++
>   scripts/lib/kdoc/kdoc_parser.py | 1670 +++++++++++++++++++++
>   scripts/lib/kdoc/kdoc_re.py     |  270 ++++
>   12 files changed, 3355 insertions(+), 2451 deletions(-)
>   delete mode 100755 scripts/kernel-doc
>   create mode 100755 scripts/kernel-doc.py
>   create mode 100644 scripts/lib/kdoc/kdoc_files.py
>   create mode 100644 scripts/lib/kdoc/kdoc_item.py
>   create mode 100644 scripts/lib/kdoc/kdoc_output.py
>   create mode 100644 scripts/lib/kdoc/kdoc_parser.py
>   create mode 100644 scripts/lib/kdoc/kdoc_re.py
>