[PATCH 0/8] Yet another set of kerneldoc simplifications

[PATCH 0/8] Yet another set of kerneldoc simplifications

[Jonathan Corbet]

As I continue to work through our shiny new kerneldoc, I keep finding ways
to make it (IMO) shinier.  This set covers these basic areas:

- Remove some unused fields from the KernelEntry class, and encapsulate the
  handling of the section contentions therein.

- Clean up and optimize the EXPORT_SYMBOL processing slightly.

- Rework the handling of inline comments by getting rid of the substate
  design and separating out the processing of the states that remain.

The series results in no changes in the generated output.

Jonathan Corbet (8):
  docs: kdoc: remove KernelEntry::in_doc_sect
  docs: kdoc: Move content handling into KernelEntry
  docs: kdoc: remove a bit of dead code
  docs: kdoc: remove KernelEntry::function
  docs: kdoc: rework process_export() slightly
  docs: kdoc: remove the INLINE_END state
  docs: kdoc: remove the inline states-within-a-state
  docs: kdoc: split the processing of the two remaining inline states

 scripts/lib/kdoc/kdoc_parser.py | 170 +++++++++++++-------------------
 1 file changed, 67 insertions(+), 103 deletions(-)

-- 
2.49.0

[PATCH 1/8] docs: kdoc: remove KernelEntry::in_doc_sect

[Jonathan Corbet]

This field is not used for anything, just get rid of it.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 3557c512c85a..f3970ffbf402 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -151,8 +151,6 @@ class KernelEntry:
 
         # State flags
         self.brcount = 0
-
-        self.in_doc_sect = False
         self.declaration_start_line = ln + 1
 
     # TODO: rename to emit_message after removal of kernel-doc.pl
@@ -1227,7 +1225,6 @@ class KernelDoc:
 
         # start a new entry
         self.reset_state(ln)
-        self.entry.in_doc_sect = False
 
         # next line is always the function name
         self.state = state.NAME
@@ -1315,7 +1312,6 @@ class KernelDoc:
     #
     def is_new_section(self, ln, line):
         if doc_sect.search(line):
-            self.entry.in_doc_sect = True
             self.state = state.BODY
             #
             # Pick out the name of our new section, tweaking it if need be.
-- 
2.49.0

[PATCH 2/8] docs: kdoc: Move content handling into KernelEntry

[Jonathan Corbet]

Rather than having other code mucking around with this bit of internal
state, encapsulate it internally.  Accumulate the description as a list of
strings, joining them at the end, which is a more efficient way of building
the text.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 62 ++++++++++++++++-----------------
 1 file changed, 30 insertions(+), 32 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index f3970ffbf402..f87355b63c19 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -128,7 +128,7 @@ class KernelEntry:
     def __init__(self, config, ln):
         self.config = config
 
-        self.contents = ""
+        self._contents = []
         self.function = ""
         self.sectcheck = ""
         self.struct_actual = ""
@@ -153,6 +153,15 @@ class KernelEntry:
         self.brcount = 0
         self.declaration_start_line = ln + 1
 
+    #
+    # Management of section contents
+    #
+    def add_text(self, text):
+        self._contents.append(text)
+
+    def contents(self):
+        return '\n'.join(self._contents) + '\n'
+
     # TODO: rename to emit_message after removal of kernel-doc.pl
     def emit_msg(self, log_msg, warning=True):
         """Emit a message"""
@@ -180,9 +189,14 @@ class KernelEntry:
         """
         Dumps section contents to arrays/hashes intended for that purpose.
         """
-
+        #
+        # If we have accumulated no contents in the default ("description")
+        # section, don't bother.
+        #
+        if self.section == SECTION_DEFAULT and not self._contents:
+            return
         name = self.section
-        contents = self.contents
+        contents = self.contents()
 
         if type_param.match(name):
             name = type_param.group(1)
@@ -206,7 +220,8 @@ class KernelEntry:
                 if name != SECTION_DEFAULT:
                     self.emit_msg(self.new_start_line,
                                   f"duplicate section name '{name}'\n")
-                self.sections[name] += contents
+                # Treat as a new paragraph - add a blank line
+                self.sections[name] += '\n' + contents
             else:
                 self.sections[name] = contents
                 self.sectionlist.append(name)
@@ -217,7 +232,7 @@ class KernelEntry:
 
         if start_new:
             self.section = SECTION_DEFAULT
-            self.contents = ""
+            self._contents = []
 
 
 class KernelDoc:
@@ -1334,16 +1349,11 @@ class KernelDoc:
             newcontents = doc_sect.group(2)
             if not newcontents:
                 newcontents = ""
-
-            if self.entry.contents.strip("\n"):
-                self.dump_section()
-
+            self.dump_section()
             self.entry.begin_section(ln, newsection)
             self.entry.leading_space = None
 
-            self.entry.contents = newcontents.lstrip()
-            if self.entry.contents:
-                self.entry.contents += "\n"
+            self.entry.add_text(newcontents.lstrip())
             return True
         return False
 
@@ -1385,7 +1395,6 @@ class KernelDoc:
             #
             if cont == "":
                 self.state = state.BODY
-                self.entry.contents += "\n"  # needed?
             #
             # Otherwise we have more of the declaration section to soak up.
             #
@@ -1407,7 +1416,6 @@ class KernelDoc:
         #
         if KernRe(r"\s*\*\s*$").match(line):
             self.entry.begin_section(ln, dump = True)
-            self.entry.contents += '\n'
             self.state = state.BODY
             return
         #
@@ -1444,7 +1452,7 @@ class KernelDoc:
             #
             # Add the trimmed result to the section and we're done.
             #
-            self.entry.contents += cont[self.entry.leading_space:] + '\n'
+            self.entry.add_text(cont[self.entry.leading_space:])
         else:
             # Unknown line, ignore
             self.emit_msg(ln, f"bad line: {line}")
@@ -1458,7 +1466,7 @@ class KernelDoc:
 
         if doc_content.search(line):
             cont = doc_content.group(1)
-            self.entry.contents += cont + "\n"
+            self.entry.add_text(cont)
         else:
             # Unknown line, ignore
             self.emit_msg(ln, f"bad line: {line}")
@@ -1470,27 +1478,20 @@ class KernelDoc:
            doc_inline_sect.search(line):
             self.entry.begin_section(ln, doc_inline_sect.group(1))
 
-            self.entry.contents = doc_inline_sect.group(2).lstrip()
-            if self.entry.contents != "":
-                self.entry.contents += "\n"
-
+            self.entry.add_text(doc_inline_sect.group(2).lstrip())
             self.inline_doc_state = state.INLINE_TEXT
             # Documentation block end */
             return
 
         if doc_inline_end.search(line):
-            if self.entry.contents not in ["", "\n"]:
-                self.dump_section()
-
+            self.dump_section()
             self.state = state.PROTO
             self.inline_doc_state = state.INLINE_NA
             return
 
         if doc_content.search(line):
             if self.inline_doc_state == state.INLINE_TEXT:
-                self.entry.contents += doc_content.group(1) + "\n"
-                if not self.entry.contents.strip(" ").rstrip("\n"):
-                    self.entry.contents = ""
+                self.entry.add_text(doc_content.group(1))
 
             elif self.inline_doc_state == state.INLINE_NAME:
                 self.emit_msg(ln,
@@ -1668,11 +1669,8 @@ class KernelDoc:
 
         if doc_inline_oneline.search(line):
             self.entry.begin_section(ln, doc_inline_oneline.group(1))
-            self.entry.contents = doc_inline_oneline.group(2)
-
-            if self.entry.contents != "":
-                self.entry.contents += "\n"
-                self.dump_section(start_new=False)
+            self.entry.add_text(doc_inline_oneline.group(2))
+            self.dump_section()
 
         elif doc_inline_start.search(line):
             self.state = state.INLINE
@@ -1696,7 +1694,7 @@ class KernelDoc:
             self.reset_state(ln)
 
         elif doc_content.search(line):
-            self.entry.contents += doc_content.group(1) + "\n"
+            self.entry.add_text(doc_content.group(1))
 
     def parse_export(self):
         """
-- 
2.49.0

[PATCH 3/8] docs: kdoc: remove a bit of dead code

[Jonathan Corbet]

The type_param regex matches "@..." just fine, so the special-case branch
for that in dump_section() is never executed.  Just remove it.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index f87355b63c19..9e46cfa20978 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -207,13 +207,6 @@ class KernelEntry:
             self.sectcheck += name + " "
             self.new_start_line = 0
 
-        elif name == "@...":
-            name = "..."
-            self.parameterdescs[name] = contents
-            self.sectcheck += name + " "
-            self.parameterdesc_start_lines[name] = self.new_start_line
-            self.new_start_line = 0
-
         else:
             if name in self.sections and self.sections[name] != "":
                 # Only warn on user-specified duplicate section names
-- 
2.49.0

[PATCH 4/8] docs: kdoc: remove KernelEntry::function

[Jonathan Corbet]

This member is unused, to take it out.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 9e46cfa20978..224dea5f7c2e 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -129,7 +129,6 @@ class KernelEntry:
         self.config = config
 
         self._contents = []
-        self.function = ""
         self.sectcheck = ""
         self.struct_actual = ""
         self.prototype = ""
-- 
2.49.0

[PATCH 5/8] docs: kdoc: rework process_export() slightly

[Jonathan Corbet]

Reorganize process_export() to eliminate duplicated code, don't look for
exports in states where we don't expect them, and don't bother with normal
state-machine processing if an export declaration has been found.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 30 ++++++++++++++----------------
 1 file changed, 14 insertions(+), 16 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 224dea5f7c2e..734b908579c3 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1211,16 +1211,17 @@ class KernelDoc:
 
         if export_symbol.search(line):
             symbol = export_symbol.group(2)
-            for suffix in suffixes:
-                symbol = symbol.removesuffix(suffix)
-            function_set.add(symbol)
-            return
-
-        if export_symbol_ns.search(line):
+        elif export_symbol_ns.search(line):
             symbol = export_symbol_ns.group(2)
-            for suffix in suffixes:
-                symbol = symbol.removesuffix(suffix)
-            function_set.add(symbol)
+        else:
+            return False
+        #
+        # Found an export, trim out any special suffixes
+        #
+        for suffix in suffixes:
+            symbol = symbol.removesuffix(suffix)
+        function_set.add(symbol)
+        return True
 
     def process_normal(self, ln, line):
         """
@@ -1767,13 +1768,10 @@ class KernelDoc:
                     # it was read twice. Here, we use the already-existing
                     # loop to parse exported symbols as well.
                     #
-                    # TODO: It should be noticed that not all states are
-                    # needed here. On a future cleanup, process export only
-                    # at the states that aren't handling comment markups.
-                    self.process_export(export_table, line)
-
-                    # Hand this line to the appropriate state handler
-                    self.state_actions[self.state](self, ln, line)
+                    if (self.state != state.NORMAL) or \
+                       not self.process_export(export_table, line):
+                        # Hand this line to the appropriate state handler
+                        self.state_actions[self.state](self, ln, line)
 
         except OSError:
             self.config.log.error(f"Error: Cannot open file {self.fname}")
-- 
2.49.0

[PATCH 6/8] docs: kdoc: remove the INLINE_END state

[Jonathan Corbet]

It is never used, so just get rid of it.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 734b908579c3..03a0e44707a7 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -108,8 +108,7 @@ class state:
     INLINE_NA     = 0 # not applicable ($state != INLINE)
     INLINE_NAME   = 1 # looking for member name (@foo:)
     INLINE_TEXT   = 2 # looking for member documentation
-    INLINE_END    = 3 # done
-    INLINE_ERROR  = 4 # error - Comment without header was found.
+    INLINE_ERROR  = 3 # error - Comment without header was found.
                       # Spit a warning as it's not
                       # proper kernel-doc and ignore the rest.
 
@@ -117,7 +116,6 @@ class state:
         "",
         "_NAME",
         "_TEXT",
-        "_END",
         "_ERROR",
     ]
 
-- 
2.49.0

[PATCH 7/8] docs: kdoc: remove the inline states-within-a-state

[Jonathan Corbet]

The processing of inline kerneldoc comments is a state like the rest, but
it was implemented as a set of separate substates.  Just remove the
substate logic and make the inline states normal ones like the rest.

INLINE_ERROR was never actually used for anything, so just take it out.

No changes to the generated output.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 43 ++++++++++-----------------------
 1 file changed, 13 insertions(+), 30 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 03a0e44707a7..a931c1471fa8 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -91,7 +91,8 @@ class state:
     SPECIAL_SECTION = 4      # doc section ending with a blank line
     PROTO         = 5        # scanning prototype
     DOCBLOCK      = 6        # documentation block
-    INLINE        = 7        # gathering doc outside main block
+    INLINE_NAME   = 7        # gathering doc outside main block
+    INLINE_TEXT   = 8	     # reading the body of inline docs
 
     name = [
         "NORMAL",
@@ -101,23 +102,10 @@ class state:
         "SPECIAL_SECTION",
         "PROTO",
         "DOCBLOCK",
-        "INLINE",
+        "INLINE_NAME",
+        "INLINE_TEXT",
     ]
 
-    # Inline documentation state
-    INLINE_NA     = 0 # not applicable ($state != INLINE)
-    INLINE_NAME   = 1 # looking for member name (@foo:)
-    INLINE_TEXT   = 2 # looking for member documentation
-    INLINE_ERROR  = 3 # error - Comment without header was found.
-                      # Spit a warning as it's not
-                      # proper kernel-doc and ignore the rest.
-
-    inline_name = [
-        "",
-        "_NAME",
-        "_TEXT",
-        "_ERROR",
-    ]
 
 SECTION_DEFAULT = "Description"  # default section
 
@@ -246,7 +234,6 @@ class KernelDoc:
 
         # Initial state for the state machines
         self.state = state.NORMAL
-        self.inline_doc_state = state.INLINE_NA
 
         # Store entry currently being processed
         self.entry = None
@@ -323,7 +310,6 @@ class KernelDoc:
 
         # State flags
         self.state = state.NORMAL
-        self.inline_doc_state = state.INLINE_NA
 
     def push_parameter(self, ln, decl_type, param, dtype,
                        org_arg, declaration_name):
@@ -1465,30 +1451,28 @@ class KernelDoc:
     def process_inline(self, ln, line):
         """STATE_INLINE: docbook comments within a prototype."""
 
-        if self.inline_doc_state == state.INLINE_NAME and \
+        if self.state == state.INLINE_NAME and \
            doc_inline_sect.search(line):
             self.entry.begin_section(ln, doc_inline_sect.group(1))
 
             self.entry.add_text(doc_inline_sect.group(2).lstrip())
-            self.inline_doc_state = state.INLINE_TEXT
+            self.state = state.INLINE_TEXT
             # Documentation block end */
             return
 
         if doc_inline_end.search(line):
             self.dump_section()
             self.state = state.PROTO
-            self.inline_doc_state = state.INLINE_NA
             return
 
         if doc_content.search(line):
-            if self.inline_doc_state == state.INLINE_TEXT:
+            if self.state == state.INLINE_TEXT:
                 self.entry.add_text(doc_content.group(1))
 
-            elif self.inline_doc_state == state.INLINE_NAME:
+            elif self.state == state.INLINE_NAME:
                 self.emit_msg(ln,
                               f"Incorrect use of kernel-doc format: {line}")
-
-                self.inline_doc_state = state.INLINE_ERROR
+                self.state = state.PROTO
 
     def syscall_munge(self, ln, proto):         # pylint: disable=W0613
         """
@@ -1664,8 +1648,7 @@ class KernelDoc:
             self.dump_section()
 
         elif doc_inline_start.search(line):
-            self.state = state.INLINE
-            self.inline_doc_state = state.INLINE_NAME
+            self.state = state.INLINE_NAME
 
         elif self.entry.decl_type == 'function':
             self.process_proto_function(ln, line)
@@ -1716,7 +1699,8 @@ class KernelDoc:
         state.BODY:			process_body,
         state.DECLARATION:		process_decl,
         state.SPECIAL_SECTION:		process_special,
-        state.INLINE:			process_inline,
+        state.INLINE_NAME:		process_inline,
+        state.INLINE_TEXT:		process_inline,
         state.PROTO:			process_proto,
         state.DOCBLOCK:			process_docblock,
         }
@@ -1756,9 +1740,8 @@ class KernelDoc:
                             prev = ""
                             prev_ln = None
 
-                    self.config.log.debug("%d %s%s: %s",
+                    self.config.log.debug("%d %s: %s",
                                           ln, state.name[self.state],
-                                          state.inline_name[self.inline_doc_state],
                                           line)
 
                     # This is an optimization over the original script.
-- 
2.49.0

[PATCH 8/8] docs: kdoc: split the processing of the two remaining inline states

[Jonathan Corbet]

Now that "inline_*" are just ordinary parser states, split them into two
separate functions, getting rid of some nested conditional logic.

The original process_inline() would simply ignore lines that didn't match
any of the regexes (those lacking the initial " * " marker).  I have
preserved that behavior, but we should perhaps emit a warning instead.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 37 ++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 19 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index a931c1471fa8..93938155fce2 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1448,31 +1448,30 @@ class KernelDoc:
             # Unknown line, ignore
             self.emit_msg(ln, f"bad line: {line}")
 
-    def process_inline(self, ln, line):
-        """STATE_INLINE: docbook comments within a prototype."""
+    def process_inline_name(self, ln, line):
+        """STATE_INLINE_NAME: beginning of docbook comments within a prototype."""
 
-        if self.state == state.INLINE_NAME and \
-           doc_inline_sect.search(line):
+        if doc_inline_sect.search(line):
             self.entry.begin_section(ln, doc_inline_sect.group(1))
-
             self.entry.add_text(doc_inline_sect.group(2).lstrip())
             self.state = state.INLINE_TEXT
-            # Documentation block end */
-            return
-
-        if doc_inline_end.search(line):
+        elif doc_inline_end.search(line):
             self.dump_section()
             self.state = state.PROTO
-            return
+        elif doc_content.search(line):
+            self.emit_msg(ln, f"Incorrect use of kernel-doc format: {line}")
+            self.state = state.PROTO
+        # else ... ??
 
-        if doc_content.search(line):
-            if self.state == state.INLINE_TEXT:
-                self.entry.add_text(doc_content.group(1))
+    def process_inline_text(self, ln, line):
+        """STATE_INLINE_TEXT: docbook comments within a prototype."""
 
-            elif self.state == state.INLINE_NAME:
-                self.emit_msg(ln,
-                              f"Incorrect use of kernel-doc format: {line}")
-                self.state = state.PROTO
+        if doc_inline_end.search(line):
+            self.dump_section()
+            self.state = state.PROTO
+        elif doc_content.search(line):
+            self.entry.add_text(doc_content.group(1))
+        # else ... ??
 
     def syscall_munge(self, ln, proto):         # pylint: disable=W0613
         """
@@ -1699,8 +1698,8 @@ class KernelDoc:
         state.BODY:			process_body,
         state.DECLARATION:		process_decl,
         state.SPECIAL_SECTION:		process_special,
-        state.INLINE_NAME:		process_inline,
-        state.INLINE_TEXT:		process_inline,
+        state.INLINE_NAME:		process_inline_name,
+        state.INLINE_TEXT:		process_inline_text,
         state.PROTO:			process_proto,
         state.DOCBLOCK:			process_docblock,
         }
-- 
2.49.0

Re: [PATCH 0/8] Yet another set of kerneldoc simplifications

[Mauro Carvalho Chehab]

Em Fri, 27 Jun 2025 12:39:52 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> As I continue to work through our shiny new kerneldoc, I keep finding ways
> to make it (IMO) shinier.  This set covers these basic areas:
> 
> - Remove some unused fields from the KernelEntry class, and encapsulate the
>   handling of the section contentions therein.
> 
> - Clean up and optimize the EXPORT_SYMBOL processing slightly.
> 
> - Rework the handling of inline comments by getting rid of the substate
>   design and separating out the processing of the states that remain.
> 
> The series results in no changes in the generated output.

I looked the entire series, although I didn't test. On a visual
inspection, all changes look good to me.

Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> 
> Jonathan Corbet (8):
>   docs: kdoc: remove KernelEntry::in_doc_sect
>   docs: kdoc: Move content handling into KernelEntry
>   docs: kdoc: remove a bit of dead code
>   docs: kdoc: remove KernelEntry::function
>   docs: kdoc: rework process_export() slightly
>   docs: kdoc: remove the INLINE_END state
>   docs: kdoc: remove the inline states-within-a-state
>   docs: kdoc: split the processing of the two remaining inline states
> 
>  scripts/lib/kdoc/kdoc_parser.py | 170 +++++++++++++-------------------
>  1 file changed, 67 insertions(+), 103 deletions(-)
> 



Thanks,
Mauro