Re: [PATCH v2 00/28] kernel-doc: use a C lexical tokenizer for transforms

Mauro Carvalho Chehab
2026-03-13T09:18:04Z

Hi Jon,

On Thu, 12 Mar 2026 15:54:20 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> Also, I didn't notice any relevant change on the documentation build
> time.

After more tests, I actually noticed an issue after this changeset:

https://lore.kernel.org/linux-doc/2b957decdb6cedab4268f71a166c25b7abdb9a61.1773326442.git.mchehab+huawei@kernel.org/

Basically, a broken kernel-doc like this:

/**
* enum dmub_abm_ace_curve_type - ACE curve type.
*/
enum dmub_abm_ace_curve_type {
/**
* ACE curve as defined by the SW layer.
*/
ABM_ACE_CURVE_TYPE__SW = 0,
/**
* ACE curve as defined by the SW to HW translation interface layer.
*/
ABM_ACE_CURVE_TYPE__SW_IF = 1,
};

where the inlined markups don't have "@symbol" doesn't parse well. If
you run current kernel-doc, it would produce:

.. c:enum:: dmub_abm_ace_curve_type

ACE curve type.

.. container:: kernelindent

**Constants**

``*/ ABM_ACE_CURVE_TYPE__SW = 0``
*undescribed*

`` */ ABM_ACE_CURVE_TYPE__SW_IF = 1``
*undescribed*

Because Kernel-doc currently drops the "/**" line. My fix patch
above fixes it, but inlined comments confuse enum/struct detection.
To avoid that, we need to strip comments earlier at dump_struct and
dump_enum:

https://lore.kernel.org/linux-doc/d112804ace83e0ad8496f687977596bb7f091560.1773390831.git.mchehab+huawei@kernel.org/T/#u

After such fix, the output is now:

.. c:enum:: dmub_abm_ace_curve_type

ACE curve type.

.. container:: kernelindent

**Constants**

``ABM_ACE_CURVE_TYPE__SW``
*undescribed*

``ABM_ACE_CURVE_TYPE__SW_IF``
*undescribed*

which is the result expected when there's no proper inlined
kernel-doc markups.

Due to this issue, I ended adding a 29/28 patch on this series.

> With that regards, right now, every time a CMatch replacement
> rule takes in place, it does:
>
> for each transform:
> - tokenizes the source code;
> - handle CMatch;
> - convert tokens back to a string.
>
> A possible optimization would be to do, instead:
>
> - tokenizes source code;
> - for each transform handle CMatch;
> - convert tokens back to a string.
>
> For now, I opted not do do it, because:
>
> - too much changes on a single row;
> - docs build time is taking ~3:30 minutes, which is
> about the same time it ws taken before the changes;
> - there is a very dirty hack inside function_xforms:
> (KernRe(r"_noprof"), ""). This is meant to change
> function prototypes instead of function arguments.
>
> So, if ok for you, I would prefer to merge this one first. We can later
> optimize kdoc_parser to avoid multiple token <-> string conversions.

I did such optimization and it worked fine. So, I ended adding
a 30/28 patch at the end. With that, running kernel-doc before/after
the entire series won't have significant performance changes.

# Current approach
$ time ./scripts/kernel-doc . -man >original 2>&1

real 0m37.344s
user 0m36.447s
sys 0m0.712s

# Tokenizer running multiple times (patch 29)
$ time ./scripts/kernel-doc . -man >before 2>&1

real 1m32.427s
user 1m25.377s
sys 0m1.293s

# After optimization (patch 30)
$ time ./scripts/kernel-doc . -man >after 2>&1

real 0m47.094s
user 0m46.106s
sys 0m0.751s

10 seconds slower than before when parsing everything, which affects
make mandocs, but the time differences spent at kernel-doc parser during
make htmldocs is minimal: ir is about ~4 seconds(*):

$ run_kdoc.py -none 2>/dev/null
Checking what files are currently used on documentation...
Running kernel-doc

Elapsed time: 0:00:04.348008

(*) the slowest logic when building docs with Sphinx is inside its
RST parser code.

See the enclosed script to see how I measured the parsing time for
existing ".. kernel-doc::" markups inside Documentation.

Thanks,
Mauro

---

This is the run_kdoc.py script I'm using here to pick the same files
as make htmldocs do:

#!/bin/env python3

import os
import re
import subprocess
import sys

from datetime import datetime
from glob import glob

print("Checking what files are currently used on documentation...")

kdoc_files = set()
re_kernel_doc = re.compile(r"^\.\.\s+kernel-doc::\s*(\S+)")

for fname in glob(os.path.join(".", "**"), recursive=True):
if os.path.isfile(fname) and fname.endswith(".rst"):
with open(fname, "r", encoding="utf-8") as in_fp:
data = in_fp.read()

for line in data.split("\n"):
match = re_kernel_doc.match(line)
if match:
if os.path.isfile(match.group(1)):
kdoc_files.add(match.group(1))

if not kdoc_files:
sys.exit(f"Directory doesn't contain kernel-doc tags")

cmd = [ "./tools/docs/kernel-doc" ]
cmd += sys.argv[1:]
cmd += sorted(kdoc_files)

print("Running kernel-doc")

start_time = datetime.now()

try:
result = subprocess.run(cmd, check=True)
except subprocess.CalledProcessError as e:
print(f"kernel-doc failed: {repr(e)}")

elapsed = datetime.now() - start_time
print(f"\nElapsed time: {elapsed}")