[PATCH 0/7] Further kernel-doc tweakery

[PATCH 0/7] Further kernel-doc tweakery

[Jonathan Corbet]

This is a set of miscellaneous improvements, finishing my pass over the
first parsing pass and getting into the second ("dump_*") pass.

Jonathan Corbet (7):
  docs: kdoc: don't reinvent string.strip()
  docs: kdoc: micro-optimize KernRe
  docs: kdoc: remove the brcount floor in process_proto_type()
  docs: kdoc: rework type prototype parsing
  docs: kdoc: some tweaks to process_proto_function()
  docs: kdoc: Remove a Python 2 comment
  docs: kdoc: pretty up dump_enum()

 Documentation/sphinx/kerneldoc.py |   2 -
 scripts/lib/kdoc/kdoc_parser.py   | 150 +++++++++++++++---------------
 scripts/lib/kdoc/kdoc_re.py       |   6 +-
 3 files changed, 79 insertions(+), 79 deletions(-)

-- 
2.49.0

[PATCH 1/7] docs: kdoc: don't reinvent string.strip()

[Jonathan Corbet]

process_proto_type() and process_proto_function() reinventing the strip()
string method with a whole series of separate regexes; take all that out
and just use strip().

The previous implementation also (in process_proto_type()) removed C++
comments *after* the above dance, leaving trailing whitespace in that case;
now we do the stripping afterward.  This results in exactly one output
change: the removal of a spurious space in the definition of
BACKLIGHT_POWER_REDUCED - see
https://docs.kernel.org/gpu/backlight.html#c.backlight_properties.

I note that we are putting semicolons after #define lines that really
shouldn't be there - a task for another day.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 27 +++++----------------------
 1 file changed, 5 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 93938155fce2..d9ff2d066160 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1567,17 +1567,9 @@ class KernelDoc:
                 self.entry.prototype += r.group(1) + " "
 
         if '{' in line or ';' in line or KernRe(r'\s*#\s*define').match(line):
-            # strip comments
-            r = KernRe(r'/\*.*?\*/')
-            self.entry.prototype = r.sub('', self.entry.prototype)
-
-            # strip newlines/cr's
-            r = KernRe(r'[\r\n]+')
-            self.entry.prototype = r.sub(' ', self.entry.prototype)
-
-            # strip leading spaces
-            r = KernRe(r'^\s+')
-            self.entry.prototype = r.sub('', self.entry.prototype)
+            # strip comments and surrounding spaces
+            r = KernRe(r'/\*.*\*/')
+            self.entry.prototype = r.sub('', self.entry.prototype).strip()
 
             # Handle self.entry.prototypes for function pointers like:
             #       int (*pcs_config)(struct foo)
@@ -1600,17 +1592,8 @@ class KernelDoc:
     def process_proto_type(self, ln, line):
         """Ancillary routine to process a type"""
 
-        # Strip newlines/cr's.
-        line = KernRe(r'[\r\n]+', re.S).sub(' ', line)
-
-        # Strip leading spaces
-        line = KernRe(r'^\s+', re.S).sub('', line)
-
-        # Strip trailing spaces
-        line = KernRe(r'\s+$', re.S).sub('', line)
-
-        # Strip C99-style comments to the end of the line
-        line = KernRe(r"\/\/.*$", re.S).sub('', line)
+        # Strip C99-style comments and surrounding whitespace
+        line = KernRe(r"//.*$", re.S).sub('', line).strip()
 
         # To distinguish preprocessor directive from regular declaration later.
         if line.startswith('#'):
-- 
2.49.0

[PATCH 2/7] docs: kdoc: micro-optimize KernRe

[Jonathan Corbet]

Switch KernRe::add_regex() to a try..except block to avoid looking up each
regex twice.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_re.py | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_re.py b/scripts/lib/kdoc/kdoc_re.py
index e81695b273bf..a467cd2f160b 100644
--- a/scripts/lib/kdoc/kdoc_re.py
+++ b/scripts/lib/kdoc/kdoc_re.py
@@ -29,12 +29,10 @@ class KernRe:
         """
         Adds a new regex or re-use it from the cache.
         """
-
-        if string in re_cache:
+        try:
             self.regex = re_cache[string]
-        else:
+        except KeyError:
             self.regex = re.compile(string, flags=flags)
-
             if self.cache:
                 re_cache[string] = self.regex
 
-- 
2.49.0

[PATCH 3/7] docs: kdoc: remove the brcount floor in process_proto_type()

[Jonathan Corbet]

Putting the floor under brcount does not change the output in any way, just
remove it.

Change the termination test from ==0 to <=0 to prevent infinite loops in
case somebody does something truly wacko in the code.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index d9ff2d066160..935f2a3c4b47 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1609,9 +1609,7 @@ class KernelDoc:
                 self.entry.brcount += r.group(2).count('{')
                 self.entry.brcount -= r.group(2).count('}')
 
-                self.entry.brcount = max(self.entry.brcount, 0)
-
-                if r.group(2) == ';' and self.entry.brcount == 0:
+                if r.group(2) == ';' and self.entry.brcount <= 0:
                     self.dump_declaration(ln, self.entry.prototype)
                     self.reset_state(ln)
                     break
-- 
2.49.0

[PATCH 4/7] docs: kdoc: rework type prototype parsing

[Jonathan Corbet]

process_proto_type() is using a complex regex and a "while True" loop to
split a declaration into chunks and, in the end, count brackets.  Switch to
using a simpler regex to just do the split directly, and handle each chunk
as it comes.  The result is, IMO, easier to understand and reason about.

The old algorithm would occasionally elide the space between function
parameters; see struct rng_alg->generate(), foe example.  The only output
difference is to not elide that space, which is more correct.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 43 +++++++++++++++++++--------------
 1 file changed, 25 insertions(+), 18 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 935f2a3c4b47..61da297df623 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1594,30 +1594,37 @@ class KernelDoc:
 
         # Strip C99-style comments and surrounding whitespace
         line = KernRe(r"//.*$", re.S).sub('', line).strip()
+        if not line:
+            return # nothing to see here
 
         # To distinguish preprocessor directive from regular declaration later.
         if line.startswith('#'):
             line += ";"
-
-        r = KernRe(r'([^\{\};]*)([\{\};])(.*)')
-        while True:
-            if r.search(line):
-                if self.entry.prototype:
-                    self.entry.prototype += " "
-                self.entry.prototype += r.group(1) + r.group(2)
-
-                self.entry.brcount += r.group(2).count('{')
-                self.entry.brcount -= r.group(2).count('}')
-
-                if r.group(2) == ';' and self.entry.brcount <= 0:
+        #
+        # Split the declaration on any of { } or ;, and accumulate pieces
+        # until we hit a semicolon while not inside {brackets}
+        #
+        r = KernRe(r'(.*?)([{};])')
+        for chunk in r.split(line):
+            if chunk:  # Ignore empty matches
+                self.entry.prototype += chunk
+                #
+                # This cries out for a match statement ... someday after we can
+                # drop Python 3.9 ...
+                #
+                if chunk == '{':
+                    self.entry.brcount += 1
+                elif chunk == '}':
+                    self.entry.brcount -= 1
+                elif chunk == ';' and self.entry.brcount <= 0:
                     self.dump_declaration(ln, self.entry.prototype)
                     self.reset_state(ln)
-                    break
-
-                line = r.group(3)
-            else:
-                self.entry.prototype += line
-                break
+                    return
+        #
+        # We hit the end of the line while still in the declaration; put
+        # in a space to represent the newline.
+        #
+        self.entry.prototype += ' '
 
     def process_proto(self, ln, line):
         """STATE_PROTO: reading a function/whatever prototype."""
-- 
2.49.0

[PATCH 5/7] docs: kdoc: some tweaks to process_proto_function()

[Jonathan Corbet]

Add a set of comments to process_proto_function and reorganize the logic
slightly; no functional change.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 43 ++++++++++++++++++---------------
 1 file changed, 24 insertions(+), 19 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index 61da297df623..d5ef3ce87438 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -1553,39 +1553,44 @@ class KernelDoc:
         """Ancillary routine to process a function prototype"""
 
         # strip C99-style comments to end of line
-        r = KernRe(r"\/\/.*$", re.S)
-        line = r.sub('', line)
-
+        line = KernRe(r"\/\/.*$", re.S).sub('', line)
+        #
+        # Soak up the line's worth of prototype text, stopping at { or ; if present.
+        #
         if KernRe(r'\s*#\s*define').match(line):
             self.entry.prototype = line
-        elif line.startswith('#'):
-            # Strip other macros like #ifdef/#ifndef/#endif/...
-            pass
-        else:
+        elif not line.startswith('#'):   # skip other preprocessor stuff
             r = KernRe(r'([^\{]*)')
             if r.match(line):
                 self.entry.prototype += r.group(1) + " "
-
+        #
+        # If we now have the whole prototype, clean it up and declare victory.
+        #
         if '{' in line or ';' in line or KernRe(r'\s*#\s*define').match(line):
             # strip comments and surrounding spaces
-            r = KernRe(r'/\*.*\*/')
-            self.entry.prototype = r.sub('', self.entry.prototype).strip()
-
+            self.entry.prototype = KernRe(r'/\*.*\*/').sub('', self.entry.prototype).strip()
+            #
             # Handle self.entry.prototypes for function pointers like:
             #       int (*pcs_config)(struct foo)
-
+            # by turning it into
+            #	    int pcs_config(struct foo)
+            #
             r = KernRe(r'^(\S+\s+)\(\s*\*(\S+)\)')
             self.entry.prototype = r.sub(r'\1\2', self.entry.prototype)
-
+            #
+            # Handle special declaration syntaxes
+            #
             if 'SYSCALL_DEFINE' in self.entry.prototype:
                 self.entry.prototype = self.syscall_munge(ln,
                                                           self.entry.prototype)
-
-            r = KernRe(r'TRACE_EVENT|DEFINE_EVENT|DEFINE_SINGLE_EVENT')
-            if r.search(self.entry.prototype):
-                self.entry.prototype = self.tracepoint_munge(ln,
-                                                             self.entry.prototype)
-
+            else:
+                r = KernRe(r'TRACE_EVENT|DEFINE_EVENT|DEFINE_SINGLE_EVENT')
+                if r.search(self.entry.prototype):
+                    self.entry.prototype = self.tracepoint_munge(ln,
+                                                                 self.entry.prototype)
+            #
+            # ... and we're done
+            #
             self.dump_function(ln, self.entry.prototype)
             self.reset_state(ln)
 
-- 
2.49.0

[PATCH 6/7] docs: kdoc: Remove a Python 2 comment

[Jonathan Corbet]

We no longer support Python 2 in the docs build chain at all, so we
certainly do not need to admonish folks to keep this file working with it.

Cc: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/sphinx/kerneldoc.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
index 51a2793dc8e2..2586b4d4e494 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -25,8 +25,6 @@
 # Authors:
 #    Jani Nikula <jani.nikula@intel.com>
 #
-# Please make sure this works on both python2 and python3.
-#
 
 import codecs
 import os
-- 
2.49.0

[PATCH 7/7] docs: kdoc: pretty up dump_enum()

[Jonathan Corbet]

Add some comments to dump_enum to help the next person who has to figure
out what it is actually doing.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/lib/kdoc/kdoc_parser.py | 39 +++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 14 deletions(-)

diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
index d5ef3ce87438..50e25cf62863 100644
--- a/scripts/lib/kdoc/kdoc_parser.py
+++ b/scripts/lib/kdoc/kdoc_parser.py
@@ -860,39 +860,48 @@ class KernelDoc:
         # Strip #define macros inside enums
         proto = KernRe(r'#\s*((define|ifdef|if)\s+|endif)[^;]*;', flags=re.S).sub('', proto)
 
-        members = None
-        declaration_name = None
-
+        #
+        # Parse out the name and members of the enum.  Typedef form first.
+        #
         r = KernRe(r'typedef\s+enum\s*\{(.*)\}\s*(\w*)\s*;')
         if r.search(proto):
             declaration_name = r.group(2)
             members = r.group(1).rstrip()
+        #
+        # Failing that, look for a straight enum
+        #
         else:
             r = KernRe(r'enum\s+(\w*)\s*\{(.*)\}')
             if r.match(proto):
                 declaration_name = r.group(1)
                 members = r.group(2).rstrip()
-
-        if not members:
-            self.emit_msg(ln, f"{proto}: error: Cannot parse enum!")
-            return
-
+        #
+        # OK, this isn't going to work.
+        #
+	    else:
+                self.emit_msg(ln, f"{proto}: error: Cannot parse enum!")
+                return
+        #
+        # Make sure we found what we were expecting.
+        #
         if self.entry.identifier != declaration_name:
             if self.entry.identifier == "":
                 self.emit_msg(ln,
                               f"{proto}: wrong kernel-doc identifier on prototype")
             else:
                 self.emit_msg(ln,
-                              f"expecting prototype for enum {self.entry.identifier}. Prototype was for enum {declaration_name} instead")
+                              f"expecting prototype for enum {self.entry.identifier}. "
+                              f"Prototype was for enum {declaration_name} instead")
             return
 
         if not declaration_name:
             declaration_name = "(anonymous)"
-
+        #
+        # Parse out the name of each enum member, and verify that we
+        # have a description for it.
+        #
         member_set = set()
-
-        members = KernRe(r'\([^;]*?[\)]').sub('', members)
-
+        members = KernRe(r'\([^;)]*\)').sub('', members)
         for arg in members.split(','):
             if not arg:
                 continue
@@ -903,7 +912,9 @@ class KernelDoc:
                 self.emit_msg(ln,
                               f"Enum value '{arg}' not described in enum '{declaration_name}'")
             member_set.add(arg)
-
+        #
+        # Ensure that every described member actually exists in the enum.
+        #
         for k in self.entry.parameterdescs:
             if k not in member_set:
                 self.emit_msg(ln,
-- 
2.49.0

Re: [PATCH 6/7] docs: kdoc: Remove a Python 2 comment

[Jani Nikula]

On Tue, 01 Jul 2025, Jonathan Corbet <corbet@lwn.net> wrote:
> We no longer support Python 2 in the docs build chain at all, so we
> certainly do not need to admonish folks to keep this file working with it.

I feel old.

Acked-by: Jani Nikula <jani.nikula@intel.com>

>
> Cc: Jani Nikula <jani.nikula@intel.com>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  Documentation/sphinx/kerneldoc.py | 2 --
>  1 file changed, 2 deletions(-)
>
> diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
> index 51a2793dc8e2..2586b4d4e494 100644
> --- a/Documentation/sphinx/kerneldoc.py
> +++ b/Documentation/sphinx/kerneldoc.py
> @@ -25,8 +25,6 @@
>  # Authors:
>  #    Jani Nikula <jani.nikula@intel.com>
>  #
> -# Please make sure this works on both python2 and python3.
> -#
>  
>  import codecs
>  import os

-- 
Jani Nikula, Intel

Re: [PATCH 0/7] Further kernel-doc tweakery

[Akira Yokosawa]

Hi Jon,

On Tue,  1 Jul 2025 14:57:23 -0600, Jonathan Corbet wrote:
> This is a set of miscellaneous improvements, finishing my pass over the
> first parsing pass and getting into the second ("dump_*") pass.
> 
> Jonathan Corbet (7):
>   docs: kdoc: don't reinvent string.strip()
>   docs: kdoc: micro-optimize KernRe
>   docs: kdoc: remove the brcount floor in process_proto_type()
>   docs: kdoc: rework type prototype parsing
>   docs: kdoc: some tweaks to process_proto_function()
>   docs: kdoc: Remove a Python 2 comment
>   docs: kdoc: pretty up dump_enum()
> 
>  Documentation/sphinx/kerneldoc.py |   2 -
>  scripts/lib/kdoc/kdoc_parser.py   | 150 +++++++++++++++---------------
>  scripts/lib/kdoc/kdoc_re.py       |   6 +-
>  3 files changed, 79 insertions(+), 79 deletions(-)
> 

I just applied this set and got the error of:

---------------------------------------------------------------
  File "/<srcdir>/scripts/lib/kdoc/kdoc_parser.py", line 881
    	    else:
    ^
TabError: inconsistent use of tabs and spaces in indentation
---------------------------------------------------------------

I didn't look into individual patches, assuming it should be an easy fix
for you.

I guess it'd be better to test (and hopefully to review) other pending
series from you and Mauro ...

        Thanks, Akira

Re: [PATCH 2/7] docs: kdoc: micro-optimize KernRe

[Mauro Carvalho Chehab]

Em Tue,  1 Jul 2025 14:57:25 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Switch KernRe::add_regex() to a try..except block to avoid looking up each
> regex twice.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/lib/kdoc/kdoc_re.py | 6 ++----
>  1 file changed, 2 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_re.py b/scripts/lib/kdoc/kdoc_re.py
> index e81695b273bf..a467cd2f160b 100644
> --- a/scripts/lib/kdoc/kdoc_re.py
> +++ b/scripts/lib/kdoc/kdoc_re.py
> @@ -29,12 +29,10 @@ class KernRe:
>          """
>          Adds a new regex or re-use it from the cache.
>          """
> -
> -        if string in re_cache:
> +        try:
>              self.regex = re_cache[string]
> -        else:
> +        except KeyError:
>              self.regex = re.compile(string, flags=flags)
> -

Hmm... I opted for this particular way of checking is that I
expect that check inside a hash at dict would be faster than
letting it crash then raise an exception. 

Btw, one easy way to check how much it affects performance
(if any) would be to run it in "rogue" mode with:

	$ time ./scripts/kernel-doc.py -N .

This will run kernel-doc.py for all files at the entire Kernel
tree, only reporting problems. If you want to do changes like
this that might introduce performance regressions, I suggest
running it once, just to fill disk caches, and then run it
again before/after such changes.

Anyway, I did such measurements before/after your patch.
the difference was not relevant: just one second of difference:

original code:

real	1m20,839s
user	1m19,594s
sys	0m0,998s

after your change:

real	1m21,805s
user	1m20,612s
sys	0m0,929s

I don't mind myself to be one second slower, but this is hardly
a micro-optimization ;-)

-

Disclaimer notice: one second of difference here can be due to
some other background process on this laptop.

Regards,
Mauro

Re: [PATCH 3/7] docs: kdoc: remove the brcount floor in process_proto_type()

[Mauro Carvalho Chehab]

Em Tue,  1 Jul 2025 14:57:26 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Putting the floor under brcount does not change the output in any way, just
> remove it.
> 
> Change the termination test from ==0 to <=0 to prevent infinite loops in
> case somebody does something truly wacko in the code.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index d9ff2d066160..935f2a3c4b47 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1609,9 +1609,7 @@ class KernelDoc:
>                  self.entry.brcount += r.group(2).count('{')
>                  self.entry.brcount -= r.group(2).count('}')
>  
> -                self.entry.brcount = max(self.entry.brcount, 0)
> -
> -                if r.group(2) == ';' and self.entry.brcount == 0:
> +                if r.group(2) == ';' and self.entry.brcount <= 0:
>                      self.dump_declaration(ln, self.entry.prototype)
>                      self.reset_state(ln)
>                      break

Re: [PATCH 1/7] docs: kdoc: don't reinvent string.strip()

[Mauro Carvalho Chehab]

Em Tue,  1 Jul 2025 14:57:24 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> process_proto_type() and process_proto_function() reinventing the strip()
> string method with a whole series of separate regexes; take all that out
> and just use strip().
> 
> The previous implementation also (in process_proto_type()) removed C++
> comments *after* the above dance, leaving trailing whitespace in that case;
> now we do the stripping afterward.  This results in exactly one output
> change: the removal of a spurious space in the definition of
> BACKLIGHT_POWER_REDUCED - see
> https://docs.kernel.org/gpu/backlight.html#c.backlight_properties.
> 
> I note that we are putting semicolons after #define lines that really
> shouldn't be there - a task for another day.

Perhaps add a FIXME note for us to not forget again about this.

> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 27 +++++----------------------
>  1 file changed, 5 insertions(+), 22 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index 93938155fce2..d9ff2d066160 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1567,17 +1567,9 @@ class KernelDoc:
>                  self.entry.prototype += r.group(1) + " "
>  
>          if '{' in line or ';' in line or KernRe(r'\s*#\s*define').match(line):
> -            # strip comments
> -            r = KernRe(r'/\*.*?\*/')
> -            self.entry.prototype = r.sub('', self.entry.prototype)
> -
> -            # strip newlines/cr's
> -            r = KernRe(r'[\r\n]+')
> -            self.entry.prototype = r.sub(' ', self.entry.prototype)
> -
> -            # strip leading spaces
> -            r = KernRe(r'^\s+')
> -            self.entry.prototype = r.sub('', self.entry.prototype)
> +            # strip comments and surrounding spaces
> +            r = KernRe(r'/\*.*\*/')
> +            self.entry.prototype = r.sub('', self.entry.prototype).strip()
>  
>              # Handle self.entry.prototypes for function pointers like:
>              #       int (*pcs_config)(struct foo)
> @@ -1600,17 +1592,8 @@ class KernelDoc:
>      def process_proto_type(self, ln, line):
>          """Ancillary routine to process a type"""
>  
> -        # Strip newlines/cr's.
> -        line = KernRe(r'[\r\n]+', re.S).sub(' ', line)
> -
> -        # Strip leading spaces
> -        line = KernRe(r'^\s+', re.S).sub('', line)
> -
> -        # Strip trailing spaces
> -        line = KernRe(r'\s+$', re.S).sub('', line)
> -
> -        # Strip C99-style comments to the end of the line
> -        line = KernRe(r"\/\/.*$", re.S).sub('', line)
> +        # Strip C99-style comments and surrounding whitespace
> +        line = KernRe(r"//.*$", re.S).sub('', line).strip()
>  
>          # To distinguish preprocessor directive from regular declaration later.
>          if line.startswith('#'):

Re: [PATCH 4/7] docs: kdoc: rework type prototype parsing

[Mauro Carvalho Chehab]

Em Tue,  1 Jul 2025 14:57:27 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> process_proto_type() is using a complex regex and a "while True" loop to
> split a declaration into chunks and, in the end, count brackets.  Switch to
> using a simpler regex to just do the split directly, and handle each chunk
> as it comes.  The result is, IMO, easier to understand and reason about.
> 
> The old algorithm would occasionally elide the space between function
> parameters; see struct rng_alg->generate(), foe example.  The only output
> difference is to not elide that space, which is more correct.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 43 +++++++++++++++++++--------------
>  1 file changed, 25 insertions(+), 18 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index 935f2a3c4b47..61da297df623 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1594,30 +1594,37 @@ class KernelDoc:
>  
>          # Strip C99-style comments and surrounding whitespace
>          line = KernRe(r"//.*$", re.S).sub('', line).strip()
> +        if not line:
> +            return # nothing to see here
>  
>          # To distinguish preprocessor directive from regular declaration later.
>          if line.startswith('#'):
>              line += ";"
> -
> -        r = KernRe(r'([^\{\};]*)([\{\};])(.*)')
> -        while True:
> -            if r.search(line):
> -                if self.entry.prototype:
> -                    self.entry.prototype += " "
> -                self.entry.prototype += r.group(1) + r.group(2)
> -
> -                self.entry.brcount += r.group(2).count('{')
> -                self.entry.brcount -= r.group(2).count('}')
> -
> -                if r.group(2) == ';' and self.entry.brcount <= 0:
> +        #
> +        # Split the declaration on any of { } or ;, and accumulate pieces
> +        # until we hit a semicolon while not inside {brackets}
> +        #
> +        r = KernRe(r'(.*?)([{};])')
> +        for chunk in r.split(line):
> +            if chunk:  # Ignore empty matches
> +                self.entry.prototype += chunk
> +                #
> +                # This cries out for a match statement ... someday after we can
> +                # drop Python 3.9 ...
> +                #
> +                if chunk == '{':
> +                    self.entry.brcount += 1
> +                elif chunk == '}':
> +                    self.entry.brcount -= 1
> +                elif chunk == ';' and self.entry.brcount <= 0:
>                      self.dump_declaration(ln, self.entry.prototype)
>                      self.reset_state(ln)
> -                    break
> -
> -                line = r.group(3)
> -            else:
> -                self.entry.prototype += line
> -                break
> +                    return
> +        #
> +        # We hit the end of the line while still in the declaration; put
> +        # in a space to represent the newline.
> +        #
> +        self.entry.prototype += ' '
>  
>      def process_proto(self, ln, line):
>          """STATE_PROTO: reading a function/whatever prototype."""

Re: [PATCH 5/7] docs: kdoc: some tweaks to process_proto_function()

[Mauro Carvalho Chehab]

Em Tue,  1 Jul 2025 14:57:28 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Add a set of comments to process_proto_function and reorganize the logic
> slightly; no functional change.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

LGTM.
Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  scripts/lib/kdoc/kdoc_parser.py | 43 ++++++++++++++++++---------------
>  1 file changed, 24 insertions(+), 19 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index 61da297df623..d5ef3ce87438 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -1553,39 +1553,44 @@ class KernelDoc:
>          """Ancillary routine to process a function prototype"""
>  
>          # strip C99-style comments to end of line
> -        r = KernRe(r"\/\/.*$", re.S)
> -        line = r.sub('', line)
> -
> +        line = KernRe(r"\/\/.*$", re.S).sub('', line)
> +        #
> +        # Soak up the line's worth of prototype text, stopping at { or ; if present.
> +        #
>          if KernRe(r'\s*#\s*define').match(line):
>              self.entry.prototype = line
> -        elif line.startswith('#'):
> -            # Strip other macros like #ifdef/#ifndef/#endif/...
> -            pass
> -        else:
> +        elif not line.startswith('#'):   # skip other preprocessor stuff
>              r = KernRe(r'([^\{]*)')
>              if r.match(line):
>                  self.entry.prototype += r.group(1) + " "
> -
> +        #
> +        # If we now have the whole prototype, clean it up and declare victory.
> +        #
>          if '{' in line or ';' in line or KernRe(r'\s*#\s*define').match(line):
>              # strip comments and surrounding spaces
> -            r = KernRe(r'/\*.*\*/')
> -            self.entry.prototype = r.sub('', self.entry.prototype).strip()
> -
> +            self.entry.prototype = KernRe(r'/\*.*\*/').sub('', self.entry.prototype).strip()
> +            #
>              # Handle self.entry.prototypes for function pointers like:
>              #       int (*pcs_config)(struct foo)
> -
> +            # by turning it into
> +            #	    int pcs_config(struct foo)
> +            #
>              r = KernRe(r'^(\S+\s+)\(\s*\*(\S+)\)')
>              self.entry.prototype = r.sub(r'\1\2', self.entry.prototype)
> -
> +            #
> +            # Handle special declaration syntaxes
> +            #
>              if 'SYSCALL_DEFINE' in self.entry.prototype:
>                  self.entry.prototype = self.syscall_munge(ln,
>                                                            self.entry.prototype)
> -
> -            r = KernRe(r'TRACE_EVENT|DEFINE_EVENT|DEFINE_SINGLE_EVENT')
> -            if r.search(self.entry.prototype):
> -                self.entry.prototype = self.tracepoint_munge(ln,
> -                                                             self.entry.prototype)
> -
> +            else:
> +                r = KernRe(r'TRACE_EVENT|DEFINE_EVENT|DEFINE_SINGLE_EVENT')
> +                if r.search(self.entry.prototype):
> +                    self.entry.prototype = self.tracepoint_munge(ln,
> +                                                                 self.entry.prototype)
> +            #
> +            # ... and we're done
> +            #
>              self.dump_function(ln, self.entry.prototype)
>              self.reset_state(ln)
>  

Re: [PATCH 6/7] docs: kdoc: Remove a Python 2 comment

[Mauro Carvalho Chehab]

Em Tue,  1 Jul 2025 14:57:29 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> We no longer support Python 2 in the docs build chain at all, so we
> certainly do not need to admonish folks to keep this file working with it.
> 
> Cc: Jani Nikula <jani.nikula@intel.com>
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>

Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

> ---
>  Documentation/sphinx/kerneldoc.py | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
> index 51a2793dc8e2..2586b4d4e494 100644
> --- a/Documentation/sphinx/kerneldoc.py
> +++ b/Documentation/sphinx/kerneldoc.py
> @@ -25,8 +25,6 @@
>  # Authors:
>  #    Jani Nikula <jani.nikula@intel.com>
>  #
> -# Please make sure this works on both python2 and python3.
> -#
>  
>  import codecs
>  import os

Re: [PATCH 7/7] docs: kdoc: pretty up dump_enum()

[Mauro Carvalho Chehab]

Em Tue,  1 Jul 2025 14:57:30 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Add some comments to dump_enum to help the next person who has to figure
> out what it is actually doing.
> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  scripts/lib/kdoc/kdoc_parser.py | 39 +++++++++++++++++++++------------
>  1 file changed, 25 insertions(+), 14 deletions(-)
> 
> diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser.py
> index d5ef3ce87438..50e25cf62863 100644
> --- a/scripts/lib/kdoc/kdoc_parser.py
> +++ b/scripts/lib/kdoc/kdoc_parser.py
> @@ -860,39 +860,48 @@ class KernelDoc:
>          # Strip #define macros inside enums
>          proto = KernRe(r'#\s*((define|ifdef|if)\s+|endif)[^;]*;', flags=re.S).sub('', proto)
>  
> -        members = None
> -        declaration_name = None
> -
> +        #
> +        # Parse out the name and members of the enum.  Typedef form first.
> +        #
>          r = KernRe(r'typedef\s+enum\s*\{(.*)\}\s*(\w*)\s*;')
>          if r.search(proto):
>              declaration_name = r.group(2)
>              members = r.group(1).rstrip()
> +        #
> +        # Failing that, look for a straight enum
> +        #
>          else:
>              r = KernRe(r'enum\s+(\w*)\s*\{(.*)\}')
>              if r.match(proto):
>                  declaration_name = r.group(1)
>                  members = r.group(2).rstrip()
> -
> -        if not members:
> -            self.emit_msg(ln, f"{proto}: error: Cannot parse enum!")
> -            return
> -
> +        #
> +        # OK, this isn't going to work.
> +        #
> +	    else:
> +                self.emit_msg(ln, f"{proto}: error: Cannot parse enum!")
> +                return
> +        #
> +        # Make sure we found what we were expecting.
> +        #
>          if self.entry.identifier != declaration_name:
>              if self.entry.identifier == "":
>                  self.emit_msg(ln,
>                                f"{proto}: wrong kernel-doc identifier on prototype")
>              else:
>                  self.emit_msg(ln,
> -                              f"expecting prototype for enum {self.entry.identifier}. Prototype was for enum {declaration_name} instead")
> +                              f"expecting prototype for enum {self.entry.identifier}. "
> +                              f"Prototype was for enum {declaration_name} instead")

Even being a big one, my personal preference would be to break the long
string here, as keeping together is easier for grep, but yeah, I also
considered breaking it ;-)

>              return
>  
>          if not declaration_name:
>              declaration_name = "(anonymous)"
> -
> +        #
> +        # Parse out the name of each enum member, and verify that we
> +        # have a description for it.
> +        #
>          member_set = set()
> -
> -        members = KernRe(r'\([^;]*?[\)]').sub('', members)
> -
> +        members = KernRe(r'\([^;)]*\)').sub('', members)

I wonder why we had this "?" there... Not sure if it has any effect on
this particular regex. I *guess* not.

if the output is the same, I'm all for such change :-)

>          for arg in members.split(','):
>              if not arg:
>                  continue
> @@ -903,7 +912,9 @@ class KernelDoc:
>                  self.emit_msg(ln,
>                                f"Enum value '{arg}' not described in enum '{declaration_name}'")
>              member_set.add(arg)
> -
> +        #
> +        # Ensure that every described member actually exists in the enum.
> +        #
>          for k in self.entry.parameterdescs:
>              if k not in member_set:
>                  self.emit_msg(ln,

Either way, with or without changes on the above nitpicks:

Reviewed-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

Regards,
Mauro

Re: [PATCH 2/7] docs: kdoc: micro-optimize KernRe

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hmm... I opted for this particular way of checking is that I
> expect that check inside a hash at dict would be faster than
> letting it crash then raise an exception. 

Raising an exception is not quite a "crash" and, if the caching is doing
any good, it should be ... exceptional.  That pattern is often shown as
a better way to do conditional dict lookups, so I've tended to follow
it, even though I'm not a big fan of exceptions in general.

> Btw, one easy way to check how much it affects performance
> (if any) would be to run it in "rogue" mode with:
>
> 	$ time ./scripts/kernel-doc.py -N .
>
> This will run kernel-doc.py for all files at the entire Kernel
> tree, only reporting problems. If you want to do changes like
> this that might introduce performance regressions, I suggest
> running it once, just to fill disk caches, and then run it
> again before/after such changes.
>
> Anyway, I did such measurements before/after your patch.
> the difference was not relevant: just one second of difference:
>
> original code:
>
> real	1m20,839s
> user	1m19,594s
> sys	0m0,998s
>
> after your change:
>
> real	1m21,805s
> user	1m20,612s
> sys	0m0,929s
>
> I don't mind myself to be one second slower, but this is hardly
> a micro-optimization ;-)

Docs builds generally went slightly faster for me, but that is always a
noisy signal.

Anyway, I am not tied to this patch and can drop it.  Or I suppose I
could just redo it with .get(), which avoids both the double lookup and
the exception.

Thanks,

jon

Re: [PATCH 7/7] docs: kdoc: pretty up dump_enum()

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Em Tue,  1 Jul 2025 14:57:30 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>>                  self.emit_msg(ln,
>> -                              f"expecting prototype for enum {self.entry.identifier}. Prototype was for enum {declaration_name} instead")
>> +                              f"expecting prototype for enum {self.entry.identifier}. "
>> +                              f"Prototype was for enum {declaration_name} instead")
>
> Even being a big one, my personal preference would be to break the long
> string here, as keeping together is easier for grep, but yeah, I also
> considered breaking it ;-)

Did you mean your preference would be to *not* break it?

There's a non-greppable piece at the break point anyway, so I wasn't
anticipating making life harder for anybody there.

Thanks,

jon

Re: [PATCH 0/7] Further kernel-doc tweakery

[Jonathan Corbet]

Akira Yokosawa <akiyks@gmail.com> writes:

> Hi Jon,
>
> On Tue,  1 Jul 2025 14:57:23 -0600, Jonathan Corbet wrote:
>> This is a set of miscellaneous improvements, finishing my pass over the
>> first parsing pass and getting into the second ("dump_*") pass.
>> 
>> Jonathan Corbet (7):
>>   docs: kdoc: don't reinvent string.strip()
>>   docs: kdoc: micro-optimize KernRe
>>   docs: kdoc: remove the brcount floor in process_proto_type()
>>   docs: kdoc: rework type prototype parsing
>>   docs: kdoc: some tweaks to process_proto_function()
>>   docs: kdoc: Remove a Python 2 comment
>>   docs: kdoc: pretty up dump_enum()
>> 
>>  Documentation/sphinx/kerneldoc.py |   2 -
>>  scripts/lib/kdoc/kdoc_parser.py   | 150 +++++++++++++++---------------
>>  scripts/lib/kdoc/kdoc_re.py       |   6 +-
>>  3 files changed, 79 insertions(+), 79 deletions(-)
>> 
>
> I just applied this set and got the error of:
>
> ---------------------------------------------------------------
>   File "/<srcdir>/scripts/lib/kdoc/kdoc_parser.py", line 881
>     	    else:
>     ^
> TabError: inconsistent use of tabs and spaces in indentation
> ---------------------------------------------------------------
>
> I didn't look into individual patches, assuming it should be an easy fix
> for you.
>
> I guess it'd be better to test (and hopefully to review) other pending
> series from you and Mauro ...

Yes, sorry, I ran into that after sending ... I have no idea which weird
last-second edit introduced that error.  It worked before, honest...

I'll go ahead and send a new series shortly.

Thanks for testing!

jon

Re: [PATCH 2/7] docs: kdoc: micro-optimize KernRe

[Mauro Carvalho Chehab]

Em Thu, 03 Jul 2025 12:14:57 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Hmm... I opted for this particular way of checking is that I
> > expect that check inside a hash at dict would be faster than
> > letting it crash then raise an exception.   
> 
> Raising an exception is not quite a "crash" and, if the caching is doing
> any good, it should be ... exceptional.  That pattern is often shown as
> a better way to do conditional dict lookups, so I've tended to follow
> it, even though I'm not a big fan of exceptions in general.
> 
> > Btw, one easy way to check how much it affects performance
> > (if any) would be to run it in "rogue" mode with:
> >
> > 	$ time ./scripts/kernel-doc.py -N .
> >
> > This will run kernel-doc.py for all files at the entire Kernel
> > tree, only reporting problems. If you want to do changes like
> > this that might introduce performance regressions, I suggest
> > running it once, just to fill disk caches, and then run it
> > again before/after such changes.
> >
> > Anyway, I did such measurements before/after your patch.
> > the difference was not relevant: just one second of difference:
> >
> > original code:
> >
> > real	1m20,839s
> > user	1m19,594s
> > sys	0m0,998s
> >
> > after your change:
> >
> > real	1m21,805s
> > user	1m20,612s
> > sys	0m0,929s
> >
> > I don't mind myself to be one second slower, but this is hardly
> > a micro-optimization ;-)  
> 
> Docs builds generally went slightly faster for me, but that is always a
> noisy signal.

Maybe it is just some noise. When I ran the test, I executed the script
a couple of times just to ensure that disk cache won't be affecting it
too much. 

The advantage of running just kerneldoc without Sphinx is that we
avoid doctree cache and other things that would add too much
randomness at the build time.

> Anyway, I am not tied to this patch and can drop it.  Or I suppose I
> could just redo it with .get(), which avoids both the double lookup and
> the exception.

I'm fine with .get().

Thanks,
Mauro

Re: [PATCH 7/7] docs: kdoc: pretty up dump_enum()

[Mauro Carvalho Chehab]

Em Thu, 03 Jul 2025 12:17:42 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Em Tue,  1 Jul 2025 14:57:30 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >  
> >>                  self.emit_msg(ln,
> >> -                              f"expecting prototype for enum {self.entry.identifier}. Prototype was for enum {declaration_name} instead")
> >> +                              f"expecting prototype for enum {self.entry.identifier}. "
> >> +                              f"Prototype was for enum {declaration_name} instead")  
> >
> > Even being a big one, my personal preference would be to break the long
> > string here, as keeping together is easier for grep, but yeah, I also
> > considered breaking it ;-)  
> 
> Did you mean your preference would be to *not* break it?

What I meant is that I was in doubt myself of breaking long lines or
not... I opted to not break.

Yet, if you feel it looks better breaking it, go for it ;-)

> There's a non-greppable piece at the break point anyway, so I wasn't
> anticipating making life harder for anybody there.
> 
> Thanks,
> 
> jon



Thanks,
Mauro