Re: [PATCH for-10.2 0/8] docs: Update our kernel-doc script to the kernel's new Python one

[Jonathan Cameron via <qemu-devel@nongnu.org>]

On Thu, 14 Aug 2025 18:13:15 +0100
Peter Maydell <peter.maydell@linaro.org> wrote:

> Earlier this year, the Linux kernel's kernel-doc script was rewritten
> from the old Perl version into a shiny and hopefully more maintainable
> Python version. This commit series updates our copy of this script
> to the latest kernel version. I have tested it by comparing the
> generated HTML documentation and checking that there are no
> unexpected changes.
> 
> Luckily we are carrying very few local modifications to the Perl
> script, so this is fairly straightforward. The structure of the
> patchset is:
>  * a minor update to the kerneldoc.py Sphinx extension so it
>    will work with both old and new kernel-doc script output
>  * a fix to a doc comment markup error that I noticed while comparing
>    the HTML output from the two versions of the script
>  * import the new Python script, unmodified from the kernel's version
>    (conveniently the kernel calls it kernel-doc.py, so it doesn't
>    clash with the existing script)
>  * make the changes to that library code that correspond to the
>    two local QEMU-specific changes we carry
>  * tell sphinx to use the Python version
>  * delete the Perl script (I have put a diff of our local mods
>    to the Perl script in the commit message of this commit, for
>    posterity)
> 
> The diffstat looks big, but almost all of it is "import the
> kernel's new script that we trust and don't need to review in
> detail" and "delete the old script".

Given Mauro is somewhat active in qemu as well, +CC for information
if nothing else.

Jonathan



> 
> My immediate motivation for doing this update is that I noticed
> that the submitter of https://gitlab.com/qemu-project/qemu/-/issues/3077
> is using a Perl that complains about a construct in the perl script,
> which prompted me to check if the kernel folks had already fixed
> it, which it turned out that they had, by rewriting the whole thing :-)
> More generally, if we don't do this update, then we're effectively
> going to drift down the same path we did with checkpatch.pl, where
> we have our own version that diverges from the kernel's version
> and we have to maintain it ourselves.
> 
> We should also update the Sphinx plugin itself (i.e.
> docs/sphinx/kerneldoc.py), but because I did not need to do
> that to update the main kernel-doc script, I have left that as
> a separate todo item.
> 
> Testing
> -------
> 
> I looked at the HTML output of the old kernel-doc script versus the
> new one, using the following diff command which mechanically excludes
> a couple of "same minor change" everywhere diffs, and eyeballing the
> resulting ~150 lines of diff.
> 
> diff -w  -I '^<div class="kernelindent docutils container">$' -I '^</div>$' -I '^<p><strong>Definition</strong>' -r -u -x searchindex.js build/x86/docs-old-kerneldoc/manual build/x86/docs/manual
> 
> The HTML changes are:
> 
> (1) some paras now have ID tags, eg:
> -<p><strong>Functions operating on arrays of bits</strong></p>
> +<p id="functions-operating-on-arrays-of-bits"><strong>Functions operating on arrays of bits</strong></p>
> 
> (2) Some extra named <div>s, eg:
> +<div class="kernelindent docutils container">
>  <p><strong>Parameters</strong></p>
>  <dl class="simple">
>  <dt><code class="docutils literal notranslate"><span class="pre">long</span> <span class="pre">nr</span></code></dt><dd><p>the bit to set</p>
> @@ -144,12 +145,14 @@
>  <dt><code class="docutils literal notranslate"><span class="pre">unsigned</span> <span class="pre">long</span> <span class="pre">*addr</span></code></dt><dd><p>the address to start counting from</p>
>  </dd>
>  </dl>
> +</div>
> 
> (3) The new version correctly parses the multi-line Return: block for
> the memory_translate_iotlb() doc comment. You can see that the
> old HTML here had dt/dd markup, and it mis-renders in the HTML at
> https://www.qemu.org/docs/master/devel/memory.html#c.memory_translate_iotlb
> 
>  <p><strong>Return</strong></p>
> -<dl class="simple">
> -<dt>On success, return the MemoryRegion containing the <strong>iotlb</strong> translated</dt><dd><p>addr.  The MemoryRegion must not be
>  accessed after rcu_read_unlock.
> +<p>On success, return the MemoryRegion containing the <strong>iotlb</strong> translated
> +addr.  The MemoryRegion must not be accessed after rcu_read_unlock.
>  On failure, return NULL, setting <strong>errp</strong> with error.</p>
> -</dd>
> -</dl>
> +</div>
> 
> "Definition" sections now get output with a trailing colon:
> 
> -<p><strong>Definition</strong></p>
> +<div class="kernelindent docutils container">
> +<p><strong>Definition</strong>:</p>
> 
> This seems like it might be a bug in kernel-doc since the Parameters,
> Return, etc sections don't get the trailing colon. I don't think it's
> important enough to worry about.
> 
> thanks
> -- PMM
> 
> Peter Maydell (8):
>   docs/sphinx/kerneldoc.py: Handle new LINENO syntax
>   tests/qtest/libqtest.h: Remove stray space from doc comment
>   scripts: Import Python kerneldoc from Linux kernel
>   scripts/kernel-doc: strip QEMU_ from function definitions
>   scripts/kernel-doc: tweak for QEMU coding standards
>   scripts/kerneldoc: Switch to the Python kernel-doc script
>   scripts/kernel-doc: Delete the old Perl kernel-doc script
>   MAINTAINERS: Put kernel-doc under the "docs build machinery" section
> 
>  MAINTAINERS                     |    2 +
>  docs/conf.py                    |    4 +-
>  docs/sphinx/kerneldoc.py        |    7 +-
>  tests/qtest/libqtest.h          |    2 +-
>  .editorconfig                   |    2 +-
>  scripts/kernel-doc              | 2442 -------------------------------
>  scripts/kernel-doc.py           |  325 ++++
>  scripts/lib/kdoc/kdoc_files.py  |  291 ++++
>  scripts/lib/kdoc/kdoc_item.py   |   42 +
>  scripts/lib/kdoc/kdoc_output.py |  749 ++++++++++
>  scripts/lib/kdoc/kdoc_parser.py | 1670 +++++++++++++++++++++
>  scripts/lib/kdoc/kdoc_re.py     |  270 ++++
>  12 files changed, 3355 insertions(+), 2451 deletions(-)
>  delete mode 100755 scripts/kernel-doc
>  create mode 100755 scripts/kernel-doc.py
>  create mode 100644 scripts/lib/kdoc/kdoc_files.py
>  create mode 100644 scripts/lib/kdoc/kdoc_item.py
>  create mode 100644 scripts/lib/kdoc/kdoc_output.py
>  create mode 100644 scripts/lib/kdoc/kdoc_parser.py
>  create mode 100644 scripts/lib/kdoc/kdoc_re.py
>