[PATCH 0/9] docs: kdoc: rework the BODY* processing states

[PATCH 0/9] docs: kdoc: rework the BODY* processing states

[Jonathan Corbet]

This is a continuation of my effort to better understand the new kernel-doc
script and to clean up some of the legacy cruft brought over from the perl
version.

There were three states associated with the processing of the body of a
kerneldoc comment: BODY_MAYBE, BODY_WITH_BLANK_LINE, and BODY.
Unfortunately, these states did not actually match the state of the parser,
leading to a lot of deeply nested if-then-else code and special cases.
I've renamed the states to better reflect their meaning, and changed how
SPECIAL_SECTION, in particular, is managed.

  BODY_MAYBE becomes DECLARATION - we have seen the first line of the
  declaration that starts a kerneldoc comment, and don't know if we have
  hit the end of it or not.

  BODY_WITH_BLANK_LINE becomes SPECIAL_SECTION.  This state was used to
  recognize the sections within a comment with special rules -
  specifically, that the section ends with a blank line or the beginning of
  a new special section.  The declaration of parameters and sections like
  "Context" fit that description.  The old code recognized these sections
  at the *end*, with a lot of twisty logic; going into the proper state at
  the beginning simplifies things.  There are a few output changes, but I
  think they are all more correct.

  BODY remains BODY - a documentation section that does *not* end at a
  blank line.

To facilitate these changes, I have also begun the task of pulling more of
KernelEntry state management into the class itself; there is more to be
done on that front.

Jonathan Corbet (9):
  docs: kdoc: Make body_with_blank_line parsing more flexible
  docs: kdoc: consolidate the "begin section" logic
  docs: kdoc: separate out the handling of the declaration phase
  docs: kdoc: split out the special-section state
  docs: kdoc: coalesce the new-section handling
  docs: kdoc: rework the handling of SPECIAL_SECTION
  docs: kdoc: coalesce the end-of-comment processing
  docs: kdoc: Add some comments to process_decl()
  docs: kdoc: finish disentangling the BODY and SPECIAL_SECTION states

 scripts/lib/kdoc/kdoc_parser.py | 232 +++++++++++++++++++-------------
 1 file changed, 138 insertions(+), 94 deletions(-)

-- 
2.49.0

[PATCH 1/9] docs: kdoc: Make body_with_blank_line parsing more flexible

[Jonathan Corbet]

The regex in the BODY_WITH_BLANK_LINE case was looking for lines starting
with " * ", where exactly one space was allowed before the following text.
There are many kerneldoc comments where the authors have put multiple
spaces instead, leading to mis-formatting of the documentation.
Specifically, in this case, the description portion is associated with the
last of the parameters.

Allow multiple spaces in this context.

See, for example, synchronize_hardirq() and how its documentation is
formatted before and after the change.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 42b2e0936b72..c46e1b6a7d4b 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1309,7 +1309,7 @@ class KernelDoc:
         """
 
         if self.state == state.BODY_WITH_BLANK_LINE:
-            r = KernRe(r"\s*\*\s?\S")
+            r = KernRe(r"\s*\*\s*\S")
             if r.match(line):
                 self.dump_section()
                 self.entry.section = SECTION_DEFAULT
-- 
2.49.0

Re: [PATCH 1/9] docs: kdoc: Make body_with_blank_line parsing more flexible

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:04 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> The regex in the BODY_WITH_BLANK_LINE case was looking for lines starting
> with " * ", where exactly one space was allowed before the following text.
> There are many kerneldoc comments where the authors have put multiple
> spaces instead, leading to mis-formatting of the documentation.
> Specifically, in this case, the description portion is associated with the
> last of the parameters.
> 
> Allow multiple spaces in this context.
> 
> See, for example, synchronize_hardirq() and how its documentation is
> formatted before and after the change.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/lib/kdoc/kdoc_parser.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index 42b2e0936b72..c46e1b6a7d4b 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1309,7 +1309,7 @@ class KernelDoc:
>          """
>  
>          if self.state == state.BODY_WITH_BLANK_LINE:
> -            r = KernRe(r"\s*\*\s?\S")
> +            r = KernRe(r"\s*\*\s*\S")
>              if r.match(line):
>                  self.dump_section()
>                  self.entry.section = SECTION_DEFAULT

I was a little bit afraid that this might be causing regressions,
but after checking the output, it looks ok. 

So:

Acked-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Tested-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>


Thanks,
Mauro

[PATCH 2/9] docs: kdoc: consolidate the "begin section" logic

[Jonathan Corbet]

Pull the repeated "begin a section" logic into a single place and hide it
within the KernelEntry class.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 32 +++++++++++++++++---------------
 1 file changed, 17 insertions(+), 15 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index c46e1b6a7d4b..d29a61a06f6d 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -169,6 +169,15 @@ class KernelEntry:
         self.warnings.append(log_msg)
         return
 
+    #
+    # Begin a new section.
+    #
+    def begin_section(self, line_no, title = SECTION_DEFAULT, dump = False):
+        if dump:
+            self.dump_section(start_new = True)
+        self.section = title
+        self.new_start_line = line_no
+
     def dump_section(self, start_new=True):
         """
         Dumps section contents to arrays/hashes intended for that purpose.
@@ -1231,12 +1240,11 @@ class KernelDoc:
         # Check for a DOC: block and handle them specially.
         #
         if doc_block.search(line):
-            self.entry.new_start_line = ln
 
             if not doc_block.group(1):
-                self.entry.section = "Introduction"
+                self.entry.begin_section(ln, "Introduction")
             else:
-                self.entry.section = doc_block.group(1)
+                self.entry.begin_section(ln, doc_block.group(1))
 
             self.entry.identifier = self.entry.section
             self.state = state.DOCBLOCK
@@ -1270,8 +1278,7 @@ class KernelDoc:
             self.state = state.BODY
             self.entry.identifier = self.entry.identifier.strip(" ")
             # if there's no @param blocks need to set up default section here
-            self.entry.section = SECTION_DEFAULT
-            self.entry.new_start_line = ln + 1
+            self.entry.begin_section(ln + 1)
             #
             # Find the description portion, which *should* be there but
             # isn't always.
@@ -1312,8 +1319,7 @@ class KernelDoc:
             r = KernRe(r"\s*\*\s*\S")
             if r.match(line):
                 self.dump_section()
-                self.entry.section = SECTION_DEFAULT
-                self.entry.new_start_line = ln
+                self.entry.begin_section(ln)
                 self.entry.contents = ""
 
         if doc_sect.search(line):
@@ -1340,8 +1346,7 @@ class KernelDoc:
             if self.entry.contents.strip("\n"):
                 self.dump_section()
 
-            self.entry.new_start_line = ln
-            self.entry.section = newsection
+            self.entry.begin_section(ln, newsection)
             self.entry.leading_space = None
 
             self.entry.contents = newcontents.lstrip()
@@ -1370,9 +1375,7 @@ class KernelDoc:
 
             if cont == "":
                 if self.entry.section == self.section_context:
-                    self.dump_section()
-
-                    self.entry.new_start_line = ln
+                    self.entry.begin_section(ln, dump = True)
                     self.state = state.BODY
                 else:
                     if self.entry.section != SECTION_DEFAULT:
@@ -1427,8 +1430,7 @@ class KernelDoc:
 
         if self.inline_doc_state == state.INLINE_NAME and \
            doc_inline_sect.search(line):
-            self.entry.section = doc_inline_sect.group(1)
-            self.entry.new_start_line = ln
+            self.entry.begin_section(ln, doc_inline_sect.group(1))
 
             self.entry.contents = doc_inline_sect.group(2).lstrip()
             if self.entry.contents != "":
@@ -1627,7 +1629,7 @@ class KernelDoc:
         """STATE_PROTO: reading a function/whatever prototype."""
 
         if doc_inline_oneline.search(line):
-            self.entry.section = doc_inline_oneline.group(1)
+            self.entry.begin_section(ln, doc_inline_oneline.group(1))
             self.entry.contents = doc_inline_oneline.group(2)
 
             if self.entry.contents != "":
-- 
2.49.0

Re: [PATCH 2/9] docs: kdoc: consolidate the "begin section" logic

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:05 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Pull the repeated "begin a section" logic into a single place and hide it
> within the KernelEntry class.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

Heh, I had a code snippet similar to this one on one of my test branches ;-)

LGTM.

Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 32 +++++++++++++++++---------------
>  1 file changed, 17 insertions(+), 15 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index c46e1b6a7d4b..d29a61a06f6d 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -169,6 +169,15 @@ class KernelEntry:
>          self.warnings.append(log_msg)
>          return
>  
> +    #
> +    # Begin a new section.
> +    #
> +    def begin_section(self, line_no, title = SECTION_DEFAULT, dump = False):
> +        if dump:
> +            self.dump_section(start_new = True)
> +        self.section = title
> +        self.new_start_line = line_no
> +
>      def dump_section(self, start_new=True):
>          """
>          Dumps section contents to arrays/hashes intended for that purpose.
> @@ -1231,12 +1240,11 @@ class KernelDoc:
>          # Check for a DOC: block and handle them specially.
>          #
>          if doc_block.search(line):
> -            self.entry.new_start_line = ln
>  
>              if not doc_block.group(1):
> -                self.entry.section = "Introduction"
> +                self.entry.begin_section(ln, "Introduction")
>              else:
> -                self.entry.section = doc_block.group(1)
> +                self.entry.begin_section(ln, doc_block.group(1))
>  
>              self.entry.identifier = self.entry.section
>              self.state = state.DOCBLOCK
> @@ -1270,8 +1278,7 @@ class KernelDoc:
>              self.state = state.BODY
>              self.entry.identifier = self.entry.identifier.strip(" ")
>              # if there's no @param blocks need to set up default section here
> -            self.entry.section = SECTION_DEFAULT
> -            self.entry.new_start_line = ln + 1
> +            self.entry.begin_section(ln + 1)
>              #
>              # Find the description portion, which *should* be there but
>              # isn't always.
> @@ -1312,8 +1319,7 @@ class KernelDoc:
>              r = KernRe(r"\s*\*\s*\S")
>              if r.match(line):
>                  self.dump_section()
> -                self.entry.section = SECTION_DEFAULT
> -                self.entry.new_start_line = ln
> +                self.entry.begin_section(ln)
>                  self.entry.contents = ""
>  
>          if doc_sect.search(line):
> @@ -1340,8 +1346,7 @@ class KernelDoc:
>              if self.entry.contents.strip("\n"):
>                  self.dump_section()
>  
> -            self.entry.new_start_line = ln
> -            self.entry.section = newsection
> +            self.entry.begin_section(ln, newsection)
>              self.entry.leading_space = None
>  
>              self.entry.contents = newcontents.lstrip()
> @@ -1370,9 +1375,7 @@ class KernelDoc:
>  
>              if cont == "":
>                  if self.entry.section == self.section_context:
> -                    self.dump_section()
> -
> -                    self.entry.new_start_line = ln
> +                    self.entry.begin_section(ln, dump = True)
>                      self.state = state.BODY
>                  else:
>                      if self.entry.section != SECTION_DEFAULT:
> @@ -1427,8 +1430,7 @@ class KernelDoc:
>  
>          if self.inline_doc_state == state.INLINE_NAME and \
>             doc_inline_sect.search(line):
> -            self.entry.section = doc_inline_sect.group(1)
> -            self.entry.new_start_line = ln
> +            self.entry.begin_section(ln, doc_inline_sect.group(1))
>  
>              self.entry.contents = doc_inline_sect.group(2).lstrip()
>              if self.entry.contents != "":
> @@ -1627,7 +1629,7 @@ class KernelDoc:
>          """STATE_PROTO: reading a function/whatever prototype."""
>  
>          if doc_inline_oneline.search(line):
> -            self.entry.section = doc_inline_oneline.group(1)
> +            self.entry.begin_section(ln, doc_inline_oneline.group(1))
>              self.entry.contents = doc_inline_oneline.group(2)
>  
>              if self.entry.contents != "":



Thanks,
Mauro

[PATCH 3/9] docs: kdoc: separate out the handling of the declaration phase

[Jonathan Corbet]

The BODY_MAYBE state really describes the "we are in a declaration" state.
Rename it accordingly, and split the handling of this state out from that
of the other BODY* states.  This change introduces a fair amount of
duplicated code that will be coalesced in a later patch.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 93 +++++++++++++++++++++++++++------
 1 file changed, 78 insertions(+), 15 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index d29a61a06f6d..f1491f8c88e7 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -86,7 +86,7 @@ class state:
     # Parser states
     NORMAL        = 0        # normal code
     NAME          = 1        # looking for function name
-    BODY_MAYBE    = 2        # body - or maybe more description
+    DECLARATION   = 2        # We have seen a declaration which might not be done
     BODY          = 3        # the body of the comment
     BODY_WITH_BLANK_LINE = 4 # the body which has a blank line
     PROTO         = 5        # scanning prototype
@@ -96,7 +96,7 @@ class state:
     name = [
         "NORMAL",
         "NAME",
-        "BODY_MAYBE",
+        "DECLARATION",
         "BODY",
         "BODY_WITH_BLANK_LINE",
         "PROTO",
@@ -1287,7 +1287,7 @@ class KernelDoc:
             r = KernRe("[-:](.*)")
             if r.search(line):
                 self.entry.declaration_purpose = trim_whitespace(r.group(1))
-                self.state = state.BODY_MAYBE
+                self.state = state.DECLARATION
             else:
                 self.entry.declaration_purpose = ""
 
@@ -1310,9 +1310,82 @@ class KernelDoc:
         else:
             self.emit_msg(ln, f"Cannot find identifier on line:\n{line}")
 
+    def process_decl(self, ln, line):
+        """
+        STATE_DECLARATION: We've seen the beginning of a declaration
+        """
+        if doc_sect.search(line):
+            self.entry.in_doc_sect = True
+            newsection = doc_sect.group(1)
+
+            if newsection.lower() in ["description", "context"]:
+                newsection = newsection.title()
+
+            # Special case: @return is a section, not a param description
+            if newsection.lower() in ["@return", "@returns",
+                                      "return", "returns"]:
+                newsection = "Return"
+
+            # Perl kernel-doc has a check here for contents before sections.
+            # the logic there is always false, as in_doc_sect variable is
+            # always true. So, just don't implement Wcontents_before_sections
+
+            # .title()
+            newcontents = doc_sect.group(2)
+            if not newcontents:
+                newcontents = ""
+
+            if self.entry.contents.strip("\n"):
+                self.dump_section()
+
+            self.entry.begin_section(ln, newsection)
+            self.entry.leading_space = None
+
+            self.entry.contents = newcontents.lstrip()
+            if self.entry.contents:
+                self.entry.contents += "\n"
+
+            self.state = state.BODY
+            return
+
+        if doc_end.search(line):
+            self.dump_section()
+
+            # Look for doc_com + <text> + doc_end:
+            r = KernRe(r'\s*\*\s*[a-zA-Z_0-9:\.]+\*/')
+            if r.match(line):
+                self.emit_msg(ln, f"suspicious ending line: {line}")
+
+            self.entry.prototype = ""
+            self.entry.new_start_line = ln + 1
+
+            self.state = state.PROTO
+            return
+
+        if doc_content.search(line):
+            cont = doc_content.group(1)
+
+            if cont == "":
+                self.state = state.BODY
+                self.entry.contents += "\n"  # needed?
+
+            else:
+                # Continued declaration purpose
+                self.entry.declaration_purpose = self.entry.declaration_purpose.rstrip()
+                self.entry.declaration_purpose += " " + cont
+
+                r = KernRe(r"\s+")
+                self.entry.declaration_purpose = r.sub(' ',
+                                                       self.entry.declaration_purpose)
+            return
+
+        # Unknown line, ignore
+        self.emit_msg(ln, f"bad line: {line}")
+
+
     def process_body(self, ln, line):
         """
-        STATE_BODY and STATE_BODY_MAYBE: the bulk of a kerneldoc comment.
+        STATE_BODY: the bulk of a kerneldoc comment.
         """
 
         if self.state == state.BODY_WITH_BLANK_LINE:
@@ -1385,16 +1458,6 @@ class KernelDoc:
 
                     self.entry.contents += "\n"
 
-            elif self.state == state.BODY_MAYBE:
-
-                # Continued declaration purpose
-                self.entry.declaration_purpose = self.entry.declaration_purpose.rstrip()
-                self.entry.declaration_purpose += " " + cont
-
-                r = KernRe(r"\s+")
-                self.entry.declaration_purpose = r.sub(' ',
-                                                       self.entry.declaration_purpose)
-
             else:
                 if self.entry.section.startswith('@') or        \
                    self.entry.section == self.section_context:
@@ -1687,7 +1750,7 @@ class KernelDoc:
         state.NORMAL:			process_normal,
         state.NAME:			process_name,
         state.BODY:			process_body,
-        state.BODY_MAYBE:		process_body,
+        state.DECLARATION:		process_decl,
         state.BODY_WITH_BLANK_LINE:	process_body,
         state.INLINE:			process_inline,
         state.PROTO:			process_proto,
-- 
2.49.0

Re: [PATCH 3/9] docs: kdoc: separate out the handling of the declaration phase

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:06 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> The BODY_MAYBE state really describes the "we are in a declaration" state.
> Rename it accordingly, and split the handling of this state out from that
> of the other BODY* states.  This change introduces a fair amount of
> duplicated code that will be coalesced in a later patch.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 93 +++++++++++++++++++++++++++------
>  1 file changed, 78 insertions(+), 15 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index d29a61a06f6d..f1491f8c88e7 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -86,7 +86,7 @@ class state:
>      # Parser states
>      NORMAL        = 0        # normal code
>      NAME          = 1        # looking for function name
> -    BODY_MAYBE    = 2        # body - or maybe more description
> +    DECLARATION   = 2        # We have seen a declaration which might not be done

A way better now.

>      BODY          = 3        # the body of the comment
>      BODY_WITH_BLANK_LINE = 4 # the body which has a blank line
>      PROTO         = 5        # scanning prototype
> @@ -96,7 +96,7 @@ class state:
>      name = [
>          "NORMAL",
>          "NAME",
> -        "BODY_MAYBE",
> +        "DECLARATION",
>          "BODY",
>          "BODY_WITH_BLANK_LINE",
>          "PROTO",
> @@ -1287,7 +1287,7 @@ class KernelDoc:
>              r = KernRe("[-:](.*)")
>              if r.search(line):
>                  self.entry.declaration_purpose = trim_whitespace(r.group(1))
> -                self.state = state.BODY_MAYBE
> +                self.state = state.DECLARATION
>              else:
>                  self.entry.declaration_purpose = ""
>  
> @@ -1310,9 +1310,82 @@ class KernelDoc:
>          else:
>              self.emit_msg(ln, f"Cannot find identifier on line:\n{line}")
>  
> +    def process_decl(self, ln, line):
> +        """
> +        STATE_DECLARATION: We've seen the beginning of a declaration
> +        """
> +        if doc_sect.search(line):
> +            self.entry.in_doc_sect = True
> +            newsection = doc_sect.group(1)
> +
> +            if newsection.lower() in ["description", "context"]:
> +                newsection = newsection.title()
> +
> +            # Special case: @return is a section, not a param description
> +            if newsection.lower() in ["@return", "@returns",
> +                                      "return", "returns"]:
> +                newsection = "Return"
> +
> +            # Perl kernel-doc has a check here for contents before sections.
> +            # the logic there is always false, as in_doc_sect variable is
> +            # always true. So, just don't implement Wcontents_before_sections
> +
> +            # .title()
> +            newcontents = doc_sect.group(2)
> +            if not newcontents:
> +                newcontents = ""
> +
> +            if self.entry.contents.strip("\n"):
> +                self.dump_section()
> +
> +            self.entry.begin_section(ln, newsection)
> +            self.entry.leading_space = None
> +
> +            self.entry.contents = newcontents.lstrip()
> +            if self.entry.contents:
> +                self.entry.contents += "\n"
> +
> +            self.state = state.BODY
> +            return
> +
> +        if doc_end.search(line):
> +            self.dump_section()
> +
> +            # Look for doc_com + <text> + doc_end:
> +            r = KernRe(r'\s*\*\s*[a-zA-Z_0-9:\.]+\*/')
> +            if r.match(line):
> +                self.emit_msg(ln, f"suspicious ending line: {line}")
> +
> +            self.entry.prototype = ""
> +            self.entry.new_start_line = ln + 1
> +
> +            self.state = state.PROTO
> +            return
> +
> +        if doc_content.search(line):
> +            cont = doc_content.group(1)
> +
> +            if cont == "":
> +                self.state = state.BODY
> +                self.entry.contents += "\n"  # needed?
> +
> +            else:
> +                # Continued declaration purpose
> +                self.entry.declaration_purpose = self.entry.declaration_purpose.rstrip()
> +                self.entry.declaration_purpose += " " + cont
> +
> +                r = KernRe(r"\s+")
> +                self.entry.declaration_purpose = r.sub(' ',
> +                                                       self.entry.declaration_purpose)
> +            return
> +
> +        # Unknown line, ignore
> +        self.emit_msg(ln, f"bad line: {line}")
> +
> +
>      def process_body(self, ln, line):
>          """
> -        STATE_BODY and STATE_BODY_MAYBE: the bulk of a kerneldoc comment.
> +        STATE_BODY: the bulk of a kerneldoc comment.
>          """
>  
>          if self.state == state.BODY_WITH_BLANK_LINE:
> @@ -1385,16 +1458,6 @@ class KernelDoc:
>  
>                      self.entry.contents += "\n"
>  
> -            elif self.state == state.BODY_MAYBE:
> -
> -                # Continued declaration purpose
> -                self.entry.declaration_purpose = self.entry.declaration_purpose.rstrip()
> -                self.entry.declaration_purpose += " " + cont
> -
> -                r = KernRe(r"\s+")
> -                self.entry.declaration_purpose = r.sub(' ',
> -                                                       self.entry.declaration_purpose)
> -
>              else:
>                  if self.entry.section.startswith('@') or        \
>                     self.entry.section == self.section_context:
> @@ -1687,7 +1750,7 @@ class KernelDoc:
>          state.NORMAL:			process_normal,
>          state.NAME:			process_name,
>          state.BODY:			process_body,
> -        state.BODY_MAYBE:		process_body,
> +        state.DECLARATION:		process_decl,
>          state.BODY_WITH_BLANK_LINE:	process_body,
>          state.INLINE:			process_inline,
>          state.PROTO:			process_proto,



Thanks,
Mauro

[PATCH 4/9] docs: kdoc: split out the special-section state

[Jonathan Corbet]

The state known as BODY_WITH_BLANK_LINE really, in a convoluted way,
indicates a "special section" that is terminated by a blank line or the
beginning of a new section.  That is either "@param: desc" sections, or the
weird "context" section that plays by the same rules.

Rename the state to SPECIAL_SECTION and split its processing into a
separate function; no real changes to the logic yet.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index f1491f8c88e7..185ffe4e1469 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -88,7 +88,7 @@ class state:
     NAME          = 1        # looking for function name
     DECLARATION   = 2        # We have seen a declaration which might not be done
     BODY          = 3        # the body of the comment
-    BODY_WITH_BLANK_LINE = 4 # the body which has a blank line
+    SPECIAL_SECTION = 4      # doc section ending with a blank line
     PROTO         = 5        # scanning prototype
     DOCBLOCK      = 6        # documentation block
     INLINE        = 7        # gathering doc outside main block
@@ -98,7 +98,7 @@ class state:
         "NAME",
         "DECLARATION",
         "BODY",
-        "BODY_WITH_BLANK_LINE",
+        "SPECIAL_SECTION",
         "PROTO",
         "DOCBLOCK",
         "INLINE",
@@ -1383,18 +1383,18 @@ class KernelDoc:
         self.emit_msg(ln, f"bad line: {line}")
 
 
+    def process_special(self, ln, line):
+        """
+        STATE_SPECIAL_SECTION: a section ending with a blank line
+        """
+        if KernRe(r"\s*\*\s*\S").match(line):
+            self.entry.begin_section(ln, dump = True)
+        self.process_body(ln, line)
+
     def process_body(self, ln, line):
         """
         STATE_BODY: the bulk of a kerneldoc comment.
         """
-
-        if self.state == state.BODY_WITH_BLANK_LINE:
-            r = KernRe(r"\s*\*\s*\S")
-            if r.match(line):
-                self.dump_section()
-                self.entry.begin_section(ln)
-                self.entry.contents = ""
-
         if doc_sect.search(line):
             self.entry.in_doc_sect = True
             newsection = doc_sect.group(1)
@@ -1452,7 +1452,7 @@ class KernelDoc:
                     self.state = state.BODY
                 else:
                     if self.entry.section != SECTION_DEFAULT:
-                        self.state = state.BODY_WITH_BLANK_LINE
+                        self.state = state.SPECIAL_SECTION
                     else:
                         self.state = state.BODY
 
@@ -1751,7 +1751,7 @@ class KernelDoc:
         state.NAME:			process_name,
         state.BODY:			process_body,
         state.DECLARATION:		process_decl,
-        state.BODY_WITH_BLANK_LINE:	process_body,
+        state.SPECIAL_SECTION:		process_special,
         state.INLINE:			process_inline,
         state.PROTO:			process_proto,
         state.DOCBLOCK:			process_docblock,
-- 
2.49.0

Re: [PATCH 4/9] docs: kdoc: split out the special-section state

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:07 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> The state known as BODY_WITH_BLANK_LINE really, in a convoluted way,
> indicates a "special section" that is terminated by a blank line or the
> beginning of a new section.  That is either "@param: desc" sections, or the
> weird "context" section that plays by the same rules.
> 
> Rename the state to SPECIAL_SECTION and split its processing into a
> separate function; no real changes to the logic yet.

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/lib/kdoc/kdoc_parser.py | 24 ++++++++++++------------
>  1 file changed, 12 insertions(+), 12 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index f1491f8c88e7..185ffe4e1469 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -88,7 +88,7 @@ class state:
>      NAME          = 1        # looking for function name
>      DECLARATION   = 2        # We have seen a declaration which might not be done
>      BODY          = 3        # the body of the comment
> -    BODY_WITH_BLANK_LINE = 4 # the body which has a blank line
> +    SPECIAL_SECTION = 4      # doc section ending with a blank line
>      PROTO         = 5        # scanning prototype
>      DOCBLOCK      = 6        # documentation block
>      INLINE        = 7        # gathering doc outside main block
> @@ -98,7 +98,7 @@ class state:
>          "NAME",
>          "DECLARATION",
>          "BODY",
> -        "BODY_WITH_BLANK_LINE",
> +        "SPECIAL_SECTION",
>          "PROTO",
>          "DOCBLOCK",
>          "INLINE",
> @@ -1383,18 +1383,18 @@ class KernelDoc:
>          self.emit_msg(ln, f"bad line: {line}")
>  
>  
> +    def process_special(self, ln, line):
> +        """
> +        STATE_SPECIAL_SECTION: a section ending with a blank line
> +        """
> +        if KernRe(r"\s*\*\s*\S").match(line):
> +            self.entry.begin_section(ln, dump = True)
> +        self.process_body(ln, line)
> +
>      def process_body(self, ln, line):
>          """
>          STATE_BODY: the bulk of a kerneldoc comment.
>          """
> -
> -        if self.state == state.BODY_WITH_BLANK_LINE:
> -            r = KernRe(r"\s*\*\s*\S")
> -            if r.match(line):
> -                self.dump_section()
> -                self.entry.begin_section(ln)
> -                self.entry.contents = ""
> -
>          if doc_sect.search(line):
>              self.entry.in_doc_sect = True
>              newsection = doc_sect.group(1)
> @@ -1452,7 +1452,7 @@ class KernelDoc:
>                      self.state = state.BODY
>                  else:
>                      if self.entry.section != SECTION_DEFAULT:
> -                        self.state = state.BODY_WITH_BLANK_LINE
> +                        self.state = state.SPECIAL_SECTION
>                      else:
>                          self.state = state.BODY
>  
> @@ -1751,7 +1751,7 @@ class KernelDoc:
>          state.NAME:			process_name,
>          state.BODY:			process_body,
>          state.DECLARATION:		process_decl,
> -        state.BODY_WITH_BLANK_LINE:	process_body,
> +        state.SPECIAL_SECTION:		process_special,
>          state.INLINE:			process_inline,
>          state.PROTO:			process_proto,
>          state.DOCBLOCK:			process_docblock,



Thanks,
Mauro

[PATCH 5/9] docs: kdoc: coalesce the new-section handling

[Jonathan Corbet]

Merge the duplicated code back into a single implementation.  Code movement
only, no logic changes.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 49 +++++++++------------------------
 1 file changed, 13 insertions(+), 36 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 185ffe4e1469..a336d543e72b 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1310,10 +1310,10 @@ class KernelDoc:
         else:
             self.emit_msg(ln, f"Cannot find identifier on line:\n{line}")
 
-    def process_decl(self, ln, line):
-        """
-        STATE_DECLARATION: We've seen the beginning of a declaration
-        """
+    #
+    # Helper function to determine if a new section is being started.
+    #
+    def is_new_section(self, ln, line):
         if doc_sect.search(line):
             self.entry.in_doc_sect = True
             newsection = doc_sect.group(1)
@@ -1346,6 +1346,14 @@ class KernelDoc:
                 self.entry.contents += "\n"
 
             self.state = state.BODY
+            return True
+        return False
+
+    def process_decl(self, ln, line):
+        """
+        STATE_DECLARATION: We've seen the beginning of a declaration
+        """
+        if self.is_new_section(ln, line):
             return
 
         if doc_end.search(line):
@@ -1395,38 +1403,7 @@ class KernelDoc:
         """
         STATE_BODY: the bulk of a kerneldoc comment.
         """
-        if doc_sect.search(line):
-            self.entry.in_doc_sect = True
-            newsection = doc_sect.group(1)
-
-            if newsection.lower() in ["description", "context"]:
-                newsection = newsection.title()
-
-            # Special case: @return is a section, not a param description
-            if newsection.lower() in ["@return", "@returns",
-                                      "return", "returns"]:
-                newsection = "Return"
-
-            # Perl kernel-doc has a check here for contents before sections.
-            # the logic there is always false, as in_doc_sect variable is
-            # always true. So, just don't implement Wcontents_before_sections
-
-            # .title()
-            newcontents = doc_sect.group(2)
-            if not newcontents:
-                newcontents = ""
-
-            if self.entry.contents.strip("\n"):
-                self.dump_section()
-
-            self.entry.begin_section(ln, newsection)
-            self.entry.leading_space = None
-
-            self.entry.contents = newcontents.lstrip()
-            if self.entry.contents:
-                self.entry.contents += "\n"
-
-            self.state = state.BODY
+        if self.is_new_section(ln, line):
             return
 
         if doc_end.search(line):
-- 
2.49.0

Re: [PATCH 5/9] docs: kdoc: coalesce the new-section handling

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:08 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Merge the duplicated code back into a single implementation.  Code movement
> only, no logic changes.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 49 +++++++++------------------------
>  1 file changed, 13 insertions(+), 36 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index 185ffe4e1469..a336d543e72b 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1310,10 +1310,10 @@ class KernelDoc:
>          else:
>              self.emit_msg(ln, f"Cannot find identifier on line:\n{line}")
>  
> -    def process_decl(self, ln, line):
> -        """
> -        STATE_DECLARATION: We've seen the beginning of a declaration
> -        """
> +    #
> +    # Helper function to determine if a new section is being started.
> +    #
> +    def is_new_section(self, ln, line):
>          if doc_sect.search(line):
>              self.entry.in_doc_sect = True
>              newsection = doc_sect.group(1)
> @@ -1346,6 +1346,14 @@ class KernelDoc:
>                  self.entry.contents += "\n"
>  
>              self.state = state.BODY
> +            return True
> +        return False
> +
> +    def process_decl(self, ln, line):
> +        """
> +        STATE_DECLARATION: We've seen the beginning of a declaration
> +        """
> +        if self.is_new_section(ln, line):
>              return
>  
>          if doc_end.search(line):
> @@ -1395,38 +1403,7 @@ class KernelDoc:
>          """
>          STATE_BODY: the bulk of a kerneldoc comment.
>          """
> -        if doc_sect.search(line):
> -            self.entry.in_doc_sect = True
> -            newsection = doc_sect.group(1)
> -
> -            if newsection.lower() in ["description", "context"]:
> -                newsection = newsection.title()
> -
> -            # Special case: @return is a section, not a param description
> -            if newsection.lower() in ["@return", "@returns",
> -                                      "return", "returns"]:
> -                newsection = "Return"
> -
> -            # Perl kernel-doc has a check here for contents before sections.
> -            # the logic there is always false, as in_doc_sect variable is
> -            # always true. So, just don't implement Wcontents_before_sections
> -
> -            # .title()
> -            newcontents = doc_sect.group(2)
> -            if not newcontents:
> -                newcontents = ""
> -
> -            if self.entry.contents.strip("\n"):
> -                self.dump_section()
> -
> -            self.entry.begin_section(ln, newsection)
> -            self.entry.leading_space = None
> -
> -            self.entry.contents = newcontents.lstrip()
> -            if self.entry.contents:
> -                self.entry.contents += "\n"
> -
> -            self.state = state.BODY
> +        if self.is_new_section(ln, line):
>              return
>  
>          if doc_end.search(line):



Thanks,
Mauro

[PATCH 6/9] docs: kdoc: rework the handling of SPECIAL_SECTION

[Jonathan Corbet]

Move the recognition of this state to when we enter it, rather than when we
exit, eliminating some twisty logic along the way.

Some changes in output do result from this shift, generally for kerneldoc
comments that do not quite fit the format.  See, for example,
struct irqdomain.  As far as I can tell, the new behavior is more correct
in each case.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 48 ++++++++++++++-------------------
 1 file changed, 20 insertions(+), 28 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index a336d543e72b..5998b02ca3a0 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1316,21 +1316,25 @@ class KernelDoc:
     def is_new_section(self, ln, line):
         if doc_sect.search(line):
             self.entry.in_doc_sect = True
+            self.state = state.BODY
+            #
+            # Pick out the name of our new section, tweaking it if need be.
+            #
             newsection = doc_sect.group(1)
-
-            if newsection.lower() in ["description", "context"]:
-                newsection = newsection.title()
-
-            # Special case: @return is a section, not a param description
-            if newsection.lower() in ["@return", "@returns",
-                                      "return", "returns"]:
+            if newsection.lower() == 'description':
+                newsection = 'Description'
+            elif newsection.lower() == 'context':
+                newsection = 'Context'
+                self.state = state.SPECIAL_SECTION
+            elif newsection.lower() in ["@return", "@returns",
+                                        "return", "returns"]:
                 newsection = "Return"
-
-            # Perl kernel-doc has a check here for contents before sections.
-            # the logic there is always false, as in_doc_sect variable is
-            # always true. So, just don't implement Wcontents_before_sections
-
-            # .title()
+                self.state = state.SPECIAL_SECTION
+            elif newsection[0] == '@':
+                self.state = state.SPECIAL_SECTION
+            #
+            # Initialize the contents, and get the new section going.
+            #
             newcontents = doc_sect.group(2)
             if not newcontents:
                 newcontents = ""
@@ -1344,8 +1348,6 @@ class KernelDoc:
             self.entry.contents = newcontents.lstrip()
             if self.entry.contents:
                 self.entry.contents += "\n"
-
-            self.state = state.BODY
             return True
         return False
 
@@ -1395,8 +1397,9 @@ class KernelDoc:
         """
         STATE_SPECIAL_SECTION: a section ending with a blank line
         """
-        if KernRe(r"\s*\*\s*\S").match(line):
+        if KernRe(r"\s*\*\s*$").match(line):
             self.entry.begin_section(ln, dump = True)
+            self.state = state.BODY
         self.process_body(ln, line)
 
     def process_body(self, ln, line):
@@ -1424,20 +1427,9 @@ class KernelDoc:
             cont = doc_content.group(1)
 
             if cont == "":
-                if self.entry.section == self.section_context:
-                    self.entry.begin_section(ln, dump = True)
-                    self.state = state.BODY
-                else:
-                    if self.entry.section != SECTION_DEFAULT:
-                        self.state = state.SPECIAL_SECTION
-                    else:
-                        self.state = state.BODY
-
                     self.entry.contents += "\n"
-
             else:
-                if self.entry.section.startswith('@') or        \
-                   self.entry.section == self.section_context:
+                if self.state == state.SPECIAL_SECTION:
                     if self.entry.leading_space is None:
                         r = KernRe(r'^(\s+)')
                         if r.match(cont):
-- 
2.49.0

Re: [PATCH 6/9] docs: kdoc: rework the handling of SPECIAL_SECTION

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:09 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Move the recognition of this state to when we enter it, rather than when we
> exit, eliminating some twisty logic along the way.
> 
> Some changes in output do result from this shift, generally for kerneldoc
> comments that do not quite fit the format.  See, for example,
> struct irqdomain.  As far as I can tell, the new behavior is more correct
> in each case.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 48 ++++++++++++++-------------------
>  1 file changed, 20 insertions(+), 28 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index a336d543e72b..5998b02ca3a0 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1316,21 +1316,25 @@ class KernelDoc:
>      def is_new_section(self, ln, line):
>          if doc_sect.search(line):
>              self.entry.in_doc_sect = True
> +            self.state = state.BODY
> +            #
> +            # Pick out the name of our new section, tweaking it if need be.
> +            #
>              newsection = doc_sect.group(1)
> -
> -            if newsection.lower() in ["description", "context"]:
> -                newsection = newsection.title()
> -
> -            # Special case: @return is a section, not a param description
> -            if newsection.lower() in ["@return", "@returns",
> -                                      "return", "returns"]:
> +            if newsection.lower() == 'description':
> +                newsection = 'Description'
> +            elif newsection.lower() == 'context':
> +                newsection = 'Context'
> +                self.state = state.SPECIAL_SECTION
> +            elif newsection.lower() in ["@return", "@returns",
> +                                        "return", "returns"]:
>                  newsection = "Return"
> -
> -            # Perl kernel-doc has a check here for contents before sections.
> -            # the logic there is always false, as in_doc_sect variable is
> -            # always true. So, just don't implement Wcontents_before_sections
> -
> -            # .title()
> +                self.state = state.SPECIAL_SECTION
> +            elif newsection[0] == '@':
> +                self.state = state.SPECIAL_SECTION
> +            #
> +            # Initialize the contents, and get the new section going.
> +            #
>              newcontents = doc_sect.group(2)
>              if not newcontents:
>                  newcontents = ""
> @@ -1344,8 +1348,6 @@ class KernelDoc:
>              self.entry.contents = newcontents.lstrip()
>              if self.entry.contents:
>                  self.entry.contents += "\n"
> -
> -            self.state = state.BODY
>              return True
>          return False
>  
> @@ -1395,8 +1397,9 @@ class KernelDoc:
>          """
>          STATE_SPECIAL_SECTION: a section ending with a blank line
>          """
> -        if KernRe(r"\s*\*\s*\S").match(line):
> +        if KernRe(r"\s*\*\s*$").match(line):
>              self.entry.begin_section(ln, dump = True)
> +            self.state = state.BODY
>          self.process_body(ln, line)
>  
>      def process_body(self, ln, line):
> @@ -1424,20 +1427,9 @@ class KernelDoc:
>              cont = doc_content.group(1)
>  
>              if cont == "":
> -                if self.entry.section == self.section_context:
> -                    self.entry.begin_section(ln, dump = True)
> -                    self.state = state.BODY
> -                else:
> -                    if self.entry.section != SECTION_DEFAULT:
> -                        self.state = state.SPECIAL_SECTION
> -                    else:
> -                        self.state = state.BODY
> -
>                      self.entry.contents += "\n"
> -
>              else:
> -                if self.entry.section.startswith('@') or        \
> -                   self.entry.section == self.section_context:
> +                if self.state == state.SPECIAL_SECTION:
>                      if self.entry.leading_space is None:
>                          r = KernRe(r'^(\s+)')
>                          if r.match(cont):



Thanks,
Mauro

[PATCH 7/9] docs: kdoc: coalesce the end-of-comment processing

[Jonathan Corbet]

Separate out the end-of-comment logic into its own helper and remove the
duplicated code introduced earlier.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 36 +++++++++++++--------------------
 1 file changed, 14 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 5998b02ca3a0..f7a5b85a8ed7 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1351,13 +1351,10 @@ class KernelDoc:
             return True
         return False
 
-    def process_decl(self, ln, line):
-        """
-        STATE_DECLARATION: We've seen the beginning of a declaration
-        """
-        if self.is_new_section(ln, line):
-            return
-
+    #
+    # Helper function to detect (and effect) the end of a kerneldoc comment.
+    #
+    def is_comment_end(self, ln, line):
         if doc_end.search(line):
             self.dump_section()
 
@@ -1370,6 +1367,15 @@ class KernelDoc:
             self.entry.new_start_line = ln + 1
 
             self.state = state.PROTO
+            return True
+        return False
+
+
+    def process_decl(self, ln, line):
+        """
+        STATE_DECLARATION: We've seen the beginning of a declaration
+        """
+        if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
             return
 
         if doc_content.search(line):
@@ -1406,21 +1412,7 @@ class KernelDoc:
         """
         STATE_BODY: the bulk of a kerneldoc comment.
         """
-        if self.is_new_section(ln, line):
-            return
-
-        if doc_end.search(line):
-            self.dump_section()
-
-            # Look for doc_com + <text> + doc_end:
-            r = KernRe(r'\s*\*\s*[a-zA-Z_0-9:\.]+\*/')
-            if r.match(line):
-                self.emit_msg(ln, f"suspicious ending line: {line}")
-
-            self.entry.prototype = ""
-            self.entry.new_start_line = ln + 1
-
-            self.state = state.PROTO
+        if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
             return
 
         if doc_content.search(line):
-- 
2.49.0

Re: [PATCH 7/9] docs: kdoc: coalesce the end-of-comment processing

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:10 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Separate out the end-of-comment logic into its own helper and remove the
> duplicated code introduced earlier.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 36 +++++++++++++--------------------
>  1 file changed, 14 insertions(+), 22 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index 5998b02ca3a0..f7a5b85a8ed7 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1351,13 +1351,10 @@ class KernelDoc:
>              return True
>          return False
>  
> -    def process_decl(self, ln, line):
> -        """
> -        STATE_DECLARATION: We've seen the beginning of a declaration
> -        """
> -        if self.is_new_section(ln, line):
> -            return
> -
> +    #
> +    # Helper function to detect (and effect) the end of a kerneldoc comment.
> +    #
> +    def is_comment_end(self, ln, line):
>          if doc_end.search(line):
>              self.dump_section()
>  
> @@ -1370,6 +1367,15 @@ class KernelDoc:
>              self.entry.new_start_line = ln + 1
>  
>              self.state = state.PROTO
> +            return True
> +        return False
> +
> +
> +    def process_decl(self, ln, line):
> +        """
> +        STATE_DECLARATION: We've seen the beginning of a declaration
> +        """
> +        if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
>              return
>  
>          if doc_content.search(line):
> @@ -1406,21 +1412,7 @@ class KernelDoc:
>          """
>          STATE_BODY: the bulk of a kerneldoc comment.
>          """
> -        if self.is_new_section(ln, line):
> -            return
> -
> -        if doc_end.search(line):
> -            self.dump_section()
> -
> -            # Look for doc_com + <text> + doc_end:
> -            r = KernRe(r'\s*\*\s*[a-zA-Z_0-9:\.]+\*/')
> -            if r.match(line):
> -                self.emit_msg(ln, f"suspicious ending line: {line}")
> -
> -            self.entry.prototype = ""
> -            self.entry.new_start_line = ln + 1
> -
> -            self.state = state.PROTO
> +        if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
>              return
>  
>          if doc_content.search(line):



Thanks,
Mauro

[PATCH 8/9] docs: kdoc: Add some comments to process_decl()

[Jonathan Corbet]

Now that the function can actually fit into a human brain, add a few
comments.  While I was at it, I switched to the trim_whitespace() helper
rather than open-coding it.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 30 ++++++++++++++++--------------
 1 file changed, 16 insertions(+), 14 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index f7a5b85a8ed7..a6ee8bac378d 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1377,26 +1377,28 @@ class KernelDoc:
         """
         if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
             return
-
+        #
+        # Look for anything with the " * " line beginning.
+        #
         if doc_content.search(line):
             cont = doc_content.group(1)
-
+            #
+            # A blank line means that we have moved out of the declaration
+            # part of the comment (without any "special section" parameter
+            # descriptions).
+            #
             if cont == "":
                 self.state = state.BODY
                 self.entry.contents += "\n"  # needed?
-
+            #
+            # Otherwise we have more of the declaration section to soak up.
+            #
             else:
-                # Continued declaration purpose
-                self.entry.declaration_purpose = self.entry.declaration_purpose.rstrip()
-                self.entry.declaration_purpose += " " + cont
-
-                r = KernRe(r"\s+")
-                self.entry.declaration_purpose = r.sub(' ',
-                                                       self.entry.declaration_purpose)
-            return
-
-        # Unknown line, ignore
-        self.emit_msg(ln, f"bad line: {line}")
+                self.entry.declaration_purpose = \
+                    trim_whitespace(self.entry.declaration_purpose + ' ' + cont)
+        else:
+            # Unknown line, ignore
+            self.emit_msg(ln, f"bad line: {line}")
 
 
     def process_special(self, ln, line):
-- 
2.49.0

Re: [PATCH 8/9] docs: kdoc: Add some comments to process_decl()

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:11 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Now that the function can actually fit into a human brain, add a few
> comments.  While I was at it, I switched to the trim_whitespace() helper
> rather than open-coding it.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 30 ++++++++++++++++--------------
>  1 file changed, 16 insertions(+), 14 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index f7a5b85a8ed7..a6ee8bac378d 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1377,26 +1377,28 @@ class KernelDoc:
>          """
>          if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
>              return
> -
> +        #
> +        # Look for anything with the " * " line beginning.
> +        #
>          if doc_content.search(line):
>              cont = doc_content.group(1)
> -
> +            #
> +            # A blank line means that we have moved out of the declaration
> +            # part of the comment (without any "special section" parameter
> +            # descriptions).
> +            #
>              if cont == "":
>                  self.state = state.BODY
>                  self.entry.contents += "\n"  # needed?
> -
> +            #
> +            # Otherwise we have more of the declaration section to soak up.
> +            #
>              else:
> -                # Continued declaration purpose
> -                self.entry.declaration_purpose = self.entry.declaration_purpose.rstrip()
> -                self.entry.declaration_purpose += " " + cont
> -
> -                r = KernRe(r"\s+")
> -                self.entry.declaration_purpose = r.sub(' ',
> -                                                       self.entry.declaration_purpose)
> -            return
> -
> -        # Unknown line, ignore
> -        self.emit_msg(ln, f"bad line: {line}")
> +                self.entry.declaration_purpose = \
> +                    trim_whitespace(self.entry.declaration_purpose + ' ' + cont)
> +        else:
> +            # Unknown line, ignore
> +            self.emit_msg(ln, f"bad line: {line}")
>  
>  
>      def process_special(self, ln, line):



Thanks,
Mauro

[PATCH 9/9] docs: kdoc: finish disentangling the BODY and SPECIAL_SECTION states

[Jonathan Corbet]

Move the last SPECIAL_SECTION special case into the proper handler
function, getting rid of more if/then/else logic.  The leading-space
tracking was tightened up a bit in the move.  Add some comments describing
what is going on.

No changes to the generated output.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 80 ++++++++++++++++++++-------------
 1 file changed, 48 insertions(+), 32 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index a6ee8bac378d..3557c512c85a 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1405,10 +1405,53 @@ class KernelDoc:
         """
         STATE_SPECIAL_SECTION: a section ending with a blank line
         """
+        #
+        # If we have hit a blank line (only the " * " marker), then this
+        # section is done.
+        #
         if KernRe(r"\s*\*\s*$").match(line):
             self.entry.begin_section(ln, dump = True)
+            self.entry.contents += '\n'
             self.state = state.BODY
-        self.process_body(ln, line)
+            return
+        #
+        # Not a blank line, look for the other ways to end the section.
+        #
+        if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
+            return
+        #
+        # OK, we should have a continuation of the text for this section.
+        #
+        if doc_content.search(line):
+            cont = doc_content.group(1)
+            #
+            # If the lines of text after the first in a special section have
+            # leading white space, we need to trim it out or Sphinx will get
+            # confused.  For the second line (the None case), see what we
+            # find there and remember it.
+            #
+            if self.entry.leading_space is None:
+                r = KernRe(r'^(\s+)')
+                if r.match(cont):
+                    self.entry.leading_space = len(r.group(1))
+                else:
+                    self.entry.leading_space = 0
+            #
+            # Otherwise, before trimming any leading chars, be *sure*
+            # that they are white space.  We should maybe warn if this
+            # isn't the case.
+            #
+            for i in range(0, self.entry.leading_space):
+                if cont[i] != " ":
+                    self.entry.leading_space = i
+                    break
+            #
+            # Add the trimmed result to the section and we're done.
+            #
+            self.entry.contents += cont[self.entry.leading_space:] + '\n'
+        else:
+            # Unknown line, ignore
+            self.emit_msg(ln, f"bad line: {line}")
 
     def process_body(self, ln, line):
         """
@@ -1419,37 +1462,10 @@ class KernelDoc:
 
         if doc_content.search(line):
             cont = doc_content.group(1)
-
-            if cont == "":
-                    self.entry.contents += "\n"
-            else:
-                if self.state == state.SPECIAL_SECTION:
-                    if self.entry.leading_space is None:
-                        r = KernRe(r'^(\s+)')
-                        if r.match(cont):
-                            self.entry.leading_space = len(r.group(1))
-                        else:
-                            self.entry.leading_space = 0
-
-                    # Double-check if leading space are realy spaces
-                    pos = 0
-                    for i in range(0, self.entry.leading_space):
-                        if cont[i] != " ":
-                            break
-                        pos += 1
-
-                    cont = cont[pos:]
-
-                    # NEW LOGIC:
-                    # In case it is different, update it
-                    if self.entry.leading_space != pos:
-                        self.entry.leading_space = pos
-
-                self.entry.contents += cont + "\n"
-            return
-
-        # Unknown line, ignore
-        self.emit_msg(ln, f"bad line: {line}")
+            self.entry.contents += cont + "\n"
+        else:
+            # Unknown line, ignore
+            self.emit_msg(ln, f"bad line: {line}")
 
     def process_inline(self, ln, line):
         """STATE_INLINE: docbook comments within a prototype."""
-- 
2.49.0

Re: [PATCH 9/9] docs: kdoc: finish disentangling the BODY and SPECIAL_SECTION states

[Mauro Carvalho Chehab]

Em Sat, 21 Jun 2025 14:35:12 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Move the last SPECIAL_SECTION special case into the proper handler
> function, getting rid of more if/then/else logic.  The leading-space
> tracking was tightened up a bit in the move.  Add some comments describing
> what is going on.
> 
> No changes to the generated output.

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/lib/kdoc/kdoc_parser.py | 80 ++++++++++++++++++++-------------
>  1 file changed, 48 insertions(+), 32 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index a6ee8bac378d..3557c512c85a 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1405,10 +1405,53 @@ class KernelDoc:
>          """
>          STATE_SPECIAL_SECTION: a section ending with a blank line
>          """
> +        #
> +        # If we have hit a blank line (only the " * " marker), then this
> +        # section is done.
> +        #
>          if KernRe(r"\s*\*\s*$").match(line):
>              self.entry.begin_section(ln, dump = True)
> +            self.entry.contents += '\n'
>              self.state = state.BODY
> -        self.process_body(ln, line)
> +            return
> +        #
> +        # Not a blank line, look for the other ways to end the section.
> +        #
> +        if self.is_new_section(ln, line) or self.is_comment_end(ln, line):
> +            return
> +        #
> +        # OK, we should have a continuation of the text for this section.
> +        #
> +        if doc_content.search(line):
> +            cont = doc_content.group(1)
> +            #
> +            # If the lines of text after the first in a special section have
> +            # leading white space, we need to trim it out or Sphinx will get
> +            # confused.  For the second line (the None case), see what we
> +            # find there and remember it.
> +            #
> +            if self.entry.leading_space is None:
> +                r = KernRe(r'^(\s+)')
> +                if r.match(cont):
> +                    self.entry.leading_space = len(r.group(1))
> +                else:
> +                    self.entry.leading_space = 0
> +            #
> +            # Otherwise, before trimming any leading chars, be *sure*
> +            # that they are white space.  We should maybe warn if this
> +            # isn't the case.
> +            #
> +            for i in range(0, self.entry.leading_space):
> +                if cont[i] != " ":
> +                    self.entry.leading_space = i
> +                    break
> +            #
> +            # Add the trimmed result to the section and we're done.
> +            #
> +            self.entry.contents += cont[self.entry.leading_space:] + '\n'
> +        else:
> +            # Unknown line, ignore
> +            self.emit_msg(ln, f"bad line: {line}")
>  
>      def process_body(self, ln, line):
>          """
> @@ -1419,37 +1462,10 @@ class KernelDoc:
>  
>          if doc_content.search(line):
>              cont = doc_content.group(1)
> -
> -            if cont == "":
> -                    self.entry.contents += "\n"
> -            else:
> -                if self.state == state.SPECIAL_SECTION:
> -                    if self.entry.leading_space is None:
> -                        r = KernRe(r'^(\s+)')
> -                        if r.match(cont):
> -                            self.entry.leading_space = len(r.group(1))
> -                        else:
> -                            self.entry.leading_space = 0
> -
> -                    # Double-check if leading space are realy spaces
> -                    pos = 0
> -                    for i in range(0, self.entry.leading_space):
> -                        if cont[i] != " ":
> -                            break
> -                        pos += 1
> -
> -                    cont = cont[pos:]
> -
> -                    # NEW LOGIC:
> -                    # In case it is different, update it
> -                    if self.entry.leading_space != pos:
> -                        self.entry.leading_space = pos
> -
> -                self.entry.contents += cont + "\n"
> -            return
> -
> -        # Unknown line, ignore
> -        self.emit_msg(ln, f"bad line: {line}")
> +            self.entry.contents += cont + "\n"
> +        else:
> +            # Unknown line, ignore
> +            self.emit_msg(ln, f"bad line: {line}")
>  
>      def process_inline(self, ln, line):
>          """STATE_INLINE: docbook comments within a prototype."""



Thanks,
Mauro