[RFC v2 00/38] Improve ABI documentation generation

[RFC v2 00/38] Improve ABI documentation generation

[Mauro Carvalho Chehab]

Hi Jon/Greg,

That's the second version of my RFC patches meant to modenize the ABI
parser that I wrote in Perl.

I originally started it due to some issues I noticed when searching for
ABI symbols. While I could just go ahead and fix the already existing
script, I noticed that the script maintainance didn't have much care over
all those years, probably because it is easier to find Python programmers
those days.

Also, the code is complex and was not using modules or classes and
were using lots of global variables.

So, I decided to rewrite it in Python. I started with a manual conversion
for each function. Yet, to avoid future maintainership issues, I opted to
divide the code on three classes. One class for the ABI parser, another
one for regex conversion (used when checking symbols from local hardware,
and an extra class for undefined symbols check.

I opted to change the Sphinx integration gradually using different
patch sets on 4 phases:

1. minimal integration: just execute the new script on a similar way as
   the perl one;
2. the kernel_abi module was changed to import the ABI parser class,
   using it directly;
3. the logic at kernel_abi were rewritten to better integrate. As a bonus,
   Sphinx now imports ReST data symbol by symbol. That solves one of
   the issues I was discomfortable with the original approach: when lots
   of data is sent to Sphinx parser, it stops in the middle of processing.
   This was addressed in the past on a hacky way, but it could cause
   problems in the future as the number of symbols increase;
4. the ABI parser part is now called either by automarkup and kernel_abi.
   This allowed automarkup to solve ABI symbols.

Together with this series, I added some patches fixing issues and warnings
when generating documentation.

Some notes about this new RFC:

- despite being able to generate documentation cross-references with
  auto-markup, the documentation build timt didn't increase on my machine
  (it was actually 5 seconds faster);
- I rewrote some algorithms at the undefined symbol detection code. With
  that, it is now a lot faster than the previous version, at least on my desktop
  (wich has 24 CPU threads). I didn't try it on a server yet;
- the undefined parser can optionally use multiple CPUs. This sounds
  fancier than it seems: Python (up to version 3.12) is really bad with
  multi-CPU support and doesn't have real multi-thread support. Python
  3.13 has now an optional way to address that (although I didn't test yet,
  as it requires Python manual compilation). When such feature becomes
  standard, maybe the undefined symbols detection will be faster.

Btw, I'm opting to send this as an RFC because I'd like to have more
people testing and checking it. In particular, the undefined ABI check
is complex, as it requires lots of tricks to reduce the run time from hours
to seconds.
  
On this series we have:

patches 1 to 11: several bug fixes addressing issues at ABI symbols;
patch 12: a fix for scripts/documentation-file-ref-check
patches 13-15: create new script with rest and search logic and 
  minimally integrate with kernel_abi Sphinx extension(phase 1);
patches 16-19: implement phase 2: class integration (phase 2);
patch 20: fix a bug at kernel_abi: the way it splits lines is buggy;
patches  21-24: rewrite kernel_abi logic to make it simpler and more
  robust;
patches 25-27: add cross-reference support at automarkup;
patches 28-36: several ABI cleanups to cope with the improvements;
patch 37: implement undefined command;
patch 38: get rid of the old Perl script.

To make it easier to review/apply, I may end breaking the next version
on a couple of different patchsets. Still it would be nice to have more
people testing it and providing some feedback.

---
RFC v2:
  - Dropped a patch touching the perl script;
  - Implemented phases 2-4 of the script's logic;
  - Added ABI cross-references via automarkup;
  - Added support for undefined logic;
  - Added more ABI and scripts' fixes;
  - Added undefined logic;
  - Added a patch to remove the old tool.

Mauro Carvalho Chehab (38):
  docs: power: video.rst: fix a footnote reference
  docs: media: ipu3: fix two footnote references
  docs: block: ublk.rst: remove a reference from a dropped text
  docs: sphinx: remove kernellog.py file
  docs: sphinx/kernel_abi: adjust coding style
  docs: admin-guide: abi: add SPDX tags to ABI files
  ABI: sysfs-class-rfkill: fix kernelversion tags
  ABI: sysfs-bus-coresight-*: fix kernelversion tags
  ABI: sysfs-driver-dma-idxd: fix date tags
  ABI: sysfs-fs-f2fs: fix date tags
  ABI: sysfs-power: fix a what tag
  scripts/documentation-file-ref-check: don't check perl/python scripts
  scripts/get_abi.py: add a Python tool to generate ReST output
  scripts/get_abi.py: add support for symbol search
  docs: use get_abi.py for ABI generation
  scripts/get_abi.py: optimize parse_abi() function
  scripts/get_abi.py: use an interactor for ReST output
  docs: sphinx/kernel_abi: use AbiParser directly
  docs: sphinx/kernel_abi: reduce buffer usage for ABI messages
  docs: sphinx/kernel_abi: properly split lines
  scripts/get_abi.pl: Add filtering capabilities to rest output
  scripts/get_abi.pl: add support to parse ABI README file
  docs: sphinx/kernel_abi: parse ABI files only once
  docs: admin-guide/abi: split files from symbols
  docs: sphinx/automarkup: add cross-references for ABI
  docs: sphinx/kernel_abi: avoid warnings during Sphinx module init
  scripts/get_abi.py: Rename title name for ABI files
  docs: media: Allow creating cross-references for RC ABI
  docs: thunderbolt: Allow creating cross-references for ABI
  docs: arm: asymmetric-32bit: Allow creating cross-references for ABI
  docs: arm: generic-counter: Allow creating cross-references for ABI
  docs: iio: Allow creating cross-references ABI
  docs: networking: Allow creating cross-references statistics ABI
  docs: submit-checklist: Allow creating cross-references for ABI README
  docs: translations: Allow creating cross-references for ABI README
  docs: ABI: drop two duplicate symbols
  scripts/get_abi.py: add support for undefined ABIs
  scripts/get_abi.pl: drop now obsoleted script

 Documentation/ABI/removed/sysfs-class-rfkill  |    2 +-
 Documentation/ABI/stable/sysfs-class-rfkill   |   12 +-
 .../ABI/stable/sysfs-devices-system-cpu       |   10 -
 .../ABI/stable/sysfs-driver-dma-idxd          |    4 +-
 .../testing/sysfs-bus-coresight-devices-cti   |   78 +-
 .../testing/sysfs-bus-coresight-devices-tpdm  |   52 +-
 Documentation/ABI/testing/sysfs-fs-f2fs       |    4 +-
 Documentation/ABI/testing/sysfs-power         |    2 +-
 .../admin-guide/abi-obsolete-files.rst        |    7 +
 Documentation/admin-guide/abi-obsolete.rst    |    6 +-
 Documentation/admin-guide/abi-readme-file.rst |    6 +
 .../admin-guide/abi-removed-files.rst         |    7 +
 Documentation/admin-guide/abi-removed.rst     |    6 +-
 .../admin-guide/abi-stable-files.rst          |    7 +
 Documentation/admin-guide/abi-stable.rst      |    6 +-
 .../admin-guide/abi-testing-files.rst         |    7 +
 Documentation/admin-guide/abi-testing.rst     |    6 +-
 Documentation/admin-guide/abi.rst             |   17 +
 Documentation/admin-guide/media/ipu3.rst      |   12 +-
 Documentation/admin-guide/thunderbolt.rst     |    2 +-
 Documentation/arch/arm64/asymmetric-32bit.rst |    2 +-
 Documentation/block/ublk.rst                  |    2 -
 Documentation/driver-api/generic-counter.rst  |    4 +-
 Documentation/driver-api/iio/core.rst         |    2 +-
 Documentation/iio/iio_devbuf.rst              |    2 +-
 Documentation/networking/statistics.rst       |    2 +-
 Documentation/power/video.rst                 |    2 +-
 Documentation/process/submit-checklist.rst    |    2 +-
 Documentation/sphinx/automarkup.py            |   56 +
 Documentation/sphinx/kernel_abi.py            |  162 +-
 Documentation/sphinx/kerneldoc.py             |   14 +-
 Documentation/sphinx/kernellog.py             |   22 -
 Documentation/sphinx/kfigure.py               |   81 +-
 .../it_IT/process/submit-checklist.rst        |    2 +-
 .../sp_SP/process/submit-checklist.rst        |    2 +-
 .../zh_CN/process/submit-checklist.rst        |    2 +-
 .../zh_TW/process/submit-checklist.rst        |    2 +-
 .../userspace-api/media/rc/rc-sysfs-nodes.rst |    2 +-
 scripts/documentation-file-ref-check          |    2 +-
 scripts/get_abi.pl                            | 1103 -------------
 scripts/get_abi.py                            | 1437 +++++++++++++++++
 41 files changed, 1804 insertions(+), 1354 deletions(-)
 create mode 100644 Documentation/admin-guide/abi-obsolete-files.rst
 create mode 100644 Documentation/admin-guide/abi-readme-file.rst
 create mode 100644 Documentation/admin-guide/abi-removed-files.rst
 create mode 100644 Documentation/admin-guide/abi-stable-files.rst
 create mode 100644 Documentation/admin-guide/abi-testing-files.rst
 delete mode 100644 Documentation/sphinx/kernellog.py
 delete mode 100755 scripts/get_abi.pl
 create mode 100755 scripts/get_abi.py

-- 
2.48.1

[RFC v2 01/38] docs: power: video.rst: fix a footnote reference

[Mauro Carvalho Chehab]

Footnotes require an space (or a "\ ") before them, as otherwise
Sphinx/Docutils won't understand it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/power/video.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/power/video.rst b/Documentation/power/video.rst
index 337a2ba9f32f..8ab2458d1304 100644
--- a/Documentation/power/video.rst
+++ b/Documentation/power/video.rst
@@ -190,7 +190,7 @@ Toshiba Portege 3020CT		s3_mode (3)
 Toshiba Satellite 4030CDT	s3_mode (3) (S1 also works OK)
 Toshiba Satellite 4080XCDT      s3_mode (3) (S1 also works OK)
 Toshiba Satellite 4090XCDT      ??? [#f1]_
-Toshiba Satellite P10-554       s3_bios,s3_mode (4)[#f3]_
+Toshiba Satellite P10-554       s3_bios,s3_mode (4) [#f3]_
 Toshiba M30                     (2) xor X with nvidia driver using internal AGP
 Uniwill 244IIO			??? [#f1]_
 =============================== ===============================================
-- 
2.48.1

[RFC v2 02/38] docs: media: ipu3: fix two footnote references

[Mauro Carvalho Chehab]

Sphinx doesn't process references inside a code block. They need
to be before that.

Also, one of the refrences is missing a footnote origin. Add it.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/admin-guide/media/ipu3.rst | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/Documentation/admin-guide/media/ipu3.rst b/Documentation/admin-guide/media/ipu3.rst
index 83b3cd03b35c..b0d42cea5391 100644
--- a/Documentation/admin-guide/media/ipu3.rst
+++ b/Documentation/admin-guide/media/ipu3.rst
@@ -87,8 +87,8 @@ raw Bayer format that is specific to IPU3.
 Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
 2592x1944 image capture.
 
-Using the media controller APIs, the ov5670 sensor is configured to send
-frames in packed raw Bayer format to IPU3 CSI2 receiver.
+Using the media controller APIs\ [#f3]_, the ov5670 sensor is configured to
+send frames in packed raw Bayer format to IPU3 CSI2 receiver.
 
 .. code-block:: none
 
@@ -98,7 +98,7 @@ frames in packed raw Bayer format to IPU3 CSI2 receiver.
     # and that ov5670 sensor is connected to i2c bus 10 with address 0x36
     export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
 
-    # Establish the link for the media devices using media-ctl [#f3]_
+    # Establish the link for the media devices using media-ctl
     media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
 
     # Set the format for the media devices
@@ -248,9 +248,9 @@ hence "STILL" mode will need more power and memory bandwidth than "VIDEO" mode.
 TNR will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is
 running at "VIDEO" mode by default, the user can use v4l2 control
 V4L2_CID_INTEL_IPU3_MODE (currently defined in
-drivers/staging/media/ipu3/include/uapi/intel-ipu3.h) to query and set the
-running mode. For user, there is no difference for buffer queueing between the
-"VIDEO" and "STILL" mode, mandatory input and main output node should be
+drivers/staging/media/ipu3/include/uapi/intel-ipu3.h\ [#f5]_) to query and set
+the running mode. For user, there is no difference for buffer queueing between
+the "VIDEO" and "STILL" mode, mandatory input and main output node should be
 enabled and buffers need be queued, the statistics and the view-finder queues
 are optional.
 
-- 
2.48.1

[RFC v2 03/38] docs: block: ublk.rst: remove a reference from a dropped text

[Mauro Carvalho Chehab]

Commit  4093cb5a0634 ("ublk_drv: add mechanism for supporting unprivileged ublk device")
implemented support for unprivileged ublk devices. With that, the
text mentioning such issue was removed. However, the footnote
is still there.

Drop the footnote to remove a documentation build warning.

Fixes: 4093cb5a0634 ("ublk_drv: add mechanism for supporting unprivileged ublk device")
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/block/ublk.rst | 2 --
 1 file changed, 2 deletions(-)

diff --git a/Documentation/block/ublk.rst b/Documentation/block/ublk.rst
index 51665a3e6a50..1e0e7358e14a 100644
--- a/Documentation/block/ublk.rst
+++ b/Documentation/block/ublk.rst
@@ -333,6 +333,4 @@ References
 
 .. [#userspace_readme] https://github.com/ming1/ubdsrv/blob/master/README
 
-.. [#stefan] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/
-
 .. [#xiaoguang] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/
-- 
2.48.1

[RFC v2 04/38] docs: sphinx: remove kernellog.py file

[Mauro Carvalho Chehab]

In the past, there was a need for a wrapper due to different
Sphinx versions support (before Sphinx 1.6). This is long gone,
and now it is just a wrapper. Get rig of it to simplify
the code.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/kernel_abi.py |  5 +-
 Documentation/sphinx/kerneldoc.py  | 14 +++---
 Documentation/sphinx/kernellog.py  | 22 --------
 Documentation/sphinx/kfigure.py    | 81 ++++++++++++++++--------------
 4 files changed, 53 insertions(+), 69 deletions(-)
 delete mode 100644 Documentation/sphinx/kernellog.py

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 5911bd0d7965..8401562cb5d9 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -37,13 +37,13 @@ import os
 import subprocess
 import sys
 import re
-import kernellog
 
 from docutils import nodes, statemachine
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 from docutils.utils.error_reporting import ErrorString
 from sphinx.util.docutils import switch_source_input
+from sphinx.util import logging
 
 __version__  = '1.0'
 
@@ -64,6 +64,7 @@ class KernelCmd(Directive):
     optional_arguments = 2
     has_content = False
     final_argument_whitespace = True
+    logger = logging.getLogger('kernel_abi')
 
     option_spec = {
         "debug"     : directives.flag,
@@ -129,7 +130,7 @@ class KernelCmd(Directive):
             else:
                 content.append(line, f, ln)
 
-        kernellog.info(self.state.document.settings.env.app, "%s: parsed %i lines" % (fname, n))
+        self.logger.info("%s: parsed %i lines" % (fname, n))
 
         if content:
             self.do_parse(content, node)
diff --git a/Documentation/sphinx/kerneldoc.py b/Documentation/sphinx/kerneldoc.py
index ec1ddfff1863..be5b8fbf373f 100644
--- a/Documentation/sphinx/kerneldoc.py
+++ b/Documentation/sphinx/kerneldoc.py
@@ -39,7 +39,7 @@ from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 import sphinx
 from sphinx.util.docutils import switch_source_input
-import kernellog
+from sphinx.util import logging
 
 __version__  = '1.0'
 
@@ -56,6 +56,7 @@ class KernelDocDirective(Directive):
         'functions': directives.unchanged,
     }
     has_content = False
+    logger = logging.getLogger('kerneldoc')
 
     def run(self):
         env = self.state.document.settings.env
@@ -109,8 +110,7 @@ class KernelDocDirective(Directive):
         cmd += [filename]
 
         try:
-            kernellog.verbose(env.app,
-                              'calling kernel-doc \'%s\'' % (" ".join(cmd)))
+            self.logger.verbose("calling kernel-doc '%s'" % (" ".join(cmd)))
 
             p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
             out, err = p.communicate()
@@ -120,8 +120,8 @@ class KernelDocDirective(Directive):
             if p.returncode != 0:
                 sys.stderr.write(err)
 
-                kernellog.warn(env.app,
-                               'kernel-doc \'%s\' failed with return code %d' % (" ".join(cmd), p.returncode))
+                self.logger.warning("kernel-doc '%s' failed with return code %d"
+                                    % (" ".join(cmd), p.returncode))
                 return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
             elif env.config.kerneldoc_verbosity > 0:
                 sys.stderr.write(err)
@@ -148,8 +148,8 @@ class KernelDocDirective(Directive):
             return node.children
 
         except Exception as e:  # pylint: disable=W0703
-            kernellog.warn(env.app, 'kernel-doc \'%s\' processing failed with: %s' %
-                           (" ".join(cmd), str(e)))
+            self.logger.warning("kernel-doc '%s' processing failed with: %s" %
+                                (" ".join(cmd), str(e)))
             return [nodes.error(None, nodes.paragraph(text = "kernel-doc missing"))]
 
     def do_parse(self, result, node):
diff --git a/Documentation/sphinx/kernellog.py b/Documentation/sphinx/kernellog.py
deleted file mode 100644
index 0bc00c138cad..000000000000
--- a/Documentation/sphinx/kernellog.py
+++ /dev/null
@@ -1,22 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-#
-# Sphinx has deprecated its older logging interface, but the replacement
-# only goes back to 1.6.  So here's a wrapper layer to keep around for
-# as long as we support 1.4.
-#
-# We don't support 1.4 anymore, but we'll keep the wrappers around until
-# we change all the code to not use them anymore :)
-#
-import sphinx
-from sphinx.util import logging
-
-logger = logging.getLogger('kerneldoc')
-
-def warn(app, message):
-    logger.warning(message)
-
-def verbose(app, message):
-    logger.verbose(message)
-
-def info(app, message):
-    logger.info(message)
diff --git a/Documentation/sphinx/kfigure.py b/Documentation/sphinx/kfigure.py
index 97166333b727..383f9a695b08 100644
--- a/Documentation/sphinx/kfigure.py
+++ b/Documentation/sphinx/kfigure.py
@@ -59,12 +59,14 @@ from docutils.parsers.rst import directives
 from docutils.parsers.rst.directives import images
 import sphinx
 from sphinx.util.nodes import clean_astext
-import kernellog
+from sphinx.util import logging
 
 Figure = images.Figure
 
 __version__  = '1.0.0'
 
+logger = logging.getLogger('kfigure')
+
 # simple helper
 # -------------
 
@@ -170,7 +172,7 @@ def setupTools(app):
     """
     global dot_cmd, dot_Tpdf, convert_cmd, rsvg_convert_cmd   # pylint: disable=W0603
     global inkscape_cmd, inkscape_ver_one  # pylint: disable=W0603
-    kernellog.verbose(app, "kfigure: check installed tools ...")
+    logger.verbose("kfigure: check installed tools ...")
 
     dot_cmd = which('dot')
     convert_cmd = which('convert')
@@ -178,7 +180,7 @@ def setupTools(app):
     inkscape_cmd = which('inkscape')
 
     if dot_cmd:
-        kernellog.verbose(app, "use dot(1) from: " + dot_cmd)
+        logger.verbose("use dot(1) from: " + dot_cmd)
 
         try:
             dot_Thelp_list = subprocess.check_output([dot_cmd, '-Thelp'],
@@ -190,10 +192,11 @@ def setupTools(app):
         dot_Tpdf_ptn = b'pdf'
         dot_Tpdf = re.search(dot_Tpdf_ptn, dot_Thelp_list)
     else:
-        kernellog.warn(app, "dot(1) not found, for better output quality install "
-                       "graphviz from https://www.graphviz.org")
+        logger.warning(
+            "dot(1) not found, for better output quality install graphviz from https://www.graphviz.org"
+        )
     if inkscape_cmd:
-        kernellog.verbose(app, "use inkscape(1) from: " + inkscape_cmd)
+        logger.verbose("use inkscape(1) from: " + inkscape_cmd)
         inkscape_ver = subprocess.check_output([inkscape_cmd, '--version'],
                                                stderr=subprocess.DEVNULL)
         ver_one_ptn = b'Inkscape 1'
@@ -204,26 +207,27 @@ def setupTools(app):
 
     else:
         if convert_cmd:
-            kernellog.verbose(app, "use convert(1) from: " + convert_cmd)
+            logger.verbose("use convert(1) from: " + convert_cmd)
         else:
-            kernellog.verbose(app,
+            logger.verbose(
                 "Neither inkscape(1) nor convert(1) found.\n"
-                "For SVG to PDF conversion, "
-                "install either Inkscape (https://inkscape.org/) (preferred) or\n"
-                "ImageMagick (https://www.imagemagick.org)")
+                "For SVG to PDF conversion, install either Inkscape (https://inkscape.org/) (preferred) or\n"
+                "ImageMagick (https://www.imagemagick.org)"
+            )
 
         if rsvg_convert_cmd:
-            kernellog.verbose(app, "use rsvg-convert(1) from: " + rsvg_convert_cmd)
-            kernellog.verbose(app, "use 'dot -Tsvg' and rsvg-convert(1) for DOT -> PDF conversion")
+            logger.verbose("use rsvg-convert(1) from: " + rsvg_convert_cmd)
+            logger.verbose("use 'dot -Tsvg' and rsvg-convert(1) for DOT -> PDF conversion")
             dot_Tpdf = False
         else:
-            kernellog.verbose(app,
+            logger.verbose(
                 "rsvg-convert(1) not found.\n"
-                "  SVG rendering of convert(1) is done by ImageMagick-native renderer.")
+                "  SVG rendering of convert(1) is done by ImageMagick-native renderer."
+            )
             if dot_Tpdf:
-                kernellog.verbose(app, "use 'dot -Tpdf' for DOT -> PDF conversion")
+                logger.verbose("use 'dot -Tpdf' for DOT -> PDF conversion")
             else:
-                kernellog.verbose(app, "use 'dot -Tsvg' and convert(1) for DOT -> PDF conversion")
+                logger.verbose("use 'dot -Tsvg' and convert(1) for DOT -> PDF conversion")
 
 
 # integrate conversion tools
@@ -257,13 +261,12 @@ def convert_image(img_node, translator, src_fname=None):
 
     # in kernel builds, use 'make SPHINXOPTS=-v' to see verbose messages
 
-    kernellog.verbose(app, 'assert best format for: ' + img_node['uri'])
+    logger.verbose('assert best format for: ' + img_node['uri'])
 
     if in_ext == '.dot':
 
         if not dot_cmd:
-            kernellog.verbose(app,
-                              "dot from graphviz not available / include DOT raw.")
+            logger.verbose("dot from graphviz not available / include DOT raw.")
             img_node.replace_self(file2literal(src_fname))
 
         elif translator.builder.format == 'latex':
@@ -290,10 +293,11 @@ def convert_image(img_node, translator, src_fname=None):
 
         if translator.builder.format == 'latex':
             if not inkscape_cmd and convert_cmd is None:
-                kernellog.warn(app,
-                                  "no SVG to PDF conversion available / include SVG raw."
-                                  "\nIncluding large raw SVGs can cause xelatex error."
-                                  "\nInstall Inkscape (preferred) or ImageMagick.")
+                logger.warning(
+                    "no SVG to PDF conversion available / include SVG raw.\n"
+                    "Including large raw SVGs can cause xelatex error.\n"
+                    "Install Inkscape (preferred) or ImageMagick."
+                )
                 img_node.replace_self(file2literal(src_fname))
             else:
                 dst_fname = path.join(translator.builder.outdir, fname + '.pdf')
@@ -306,15 +310,14 @@ def convert_image(img_node, translator, src_fname=None):
         _name = dst_fname[len(str(translator.builder.outdir)) + 1:]
 
         if isNewer(dst_fname, src_fname):
-            kernellog.verbose(app,
-                              "convert: {out}/%s already exists and is newer" % _name)
+            logger.verbose("convert: {out}/%s already exists and is newer" % _name)
 
         else:
             ok = False
             mkdir(path.dirname(dst_fname))
 
             if in_ext == '.dot':
-                kernellog.verbose(app, 'convert DOT to: {out}/' + _name)
+                logger.verbose('convert DOT to: {out}/' + _name)
                 if translator.builder.format == 'latex' and not dot_Tpdf:
                     svg_fname = path.join(translator.builder.outdir, fname + '.svg')
                     ok1 = dot2format(app, src_fname, svg_fname)
@@ -325,7 +328,7 @@ def convert_image(img_node, translator, src_fname=None):
                     ok = dot2format(app, src_fname, dst_fname)
 
             elif in_ext == '.svg':
-                kernellog.verbose(app, 'convert SVG to: {out}/' + _name)
+                logger.verbose('convert SVG to: {out}/' + _name)
                 ok = svg2pdf(app, src_fname, dst_fname)
 
             if not ok:
@@ -354,7 +357,7 @@ def dot2format(app, dot_fname, out_fname):
     with open(out_fname, "w") as out:
         exit_code = subprocess.call(cmd, stdout = out)
         if exit_code != 0:
-            kernellog.warn(app,
+            logger.warning(
                           "Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
     return bool(exit_code == 0)
 
@@ -388,13 +391,14 @@ def svg2pdf(app, svg_fname, pdf_fname):
         pass
 
     if exit_code != 0:
-        kernellog.warn(app, "Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
+        logger.warning("Error #%d when calling: %s" %
+                            (exit_code, " ".join(cmd)))
         if warning_msg:
-            kernellog.warn(app, "Warning msg from %s: %s"
-                           % (cmd_name, str(warning_msg, 'utf-8')))
+            logger.warning( "Warning msg from %s: %s" %
+                                (cmd_name, str(warning_msg, 'utf-8')))
     elif warning_msg:
-        kernellog.verbose(app, "Warning msg from %s (likely harmless):\n%s"
-                          % (cmd_name, str(warning_msg, 'utf-8')))
+        logger.verbose("Warning msg from %s (likely harmless):\n%s" %
+                            (cmd_name, str(warning_msg, 'utf-8')))
 
     return bool(exit_code == 0)
 
@@ -418,7 +422,8 @@ def svg2pdf_by_rsvg(app, svg_fname, pdf_fname):
         # use stdout and stderr from parent
         exit_code = subprocess.call(cmd)
         if exit_code != 0:
-            kernellog.warn(app, "Error #%d when calling: %s" % (exit_code, " ".join(cmd)))
+            logger.warning("Error #%d when calling: %s" %
+                                (exit_code, " ".join(cmd)))
         ok = bool(exit_code == 0)
 
     return ok
@@ -513,15 +518,15 @@ def visit_kernel_render(self, node):
     app = self.builder.app
     srclang = node.get('srclang')
 
-    kernellog.verbose(app, 'visit kernel-render node lang: "%s"' % (srclang))
+    logger.verbose('visit kernel-render node lang: "%s"' % srclang)
 
     tmp_ext = RENDER_MARKUP_EXT.get(srclang, None)
     if tmp_ext is None:
-        kernellog.warn(app, 'kernel-render: "%s" unknown / include raw.' % (srclang))
+        logger.warning( 'kernel-render: "%s" unknown / include raw.' % srclang)
         return
 
     if not dot_cmd and tmp_ext == '.dot':
-        kernellog.verbose(app, "dot from graphviz not available / include raw.")
+        logger.verbose("dot from graphviz not available / include raw.")
         return
 
     literal_block = node[0]
-- 
2.48.1

[RFC v2 05/38] docs: sphinx/kernel_abi: adjust coding style

[Mauro Carvalho Chehab]

Make pylint and flake8 happier with this module's coding style

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/kernel_abi.py | 28 +++++++++++++---------------
 1 file changed, 13 insertions(+), 15 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 8401562cb5d9..a1887b50a900 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -32,32 +32,30 @@ u"""
 
 """
 
-import codecs
 import os
-import subprocess
-import sys
 import re
+import sys
 
-from docutils import nodes, statemachine
+from docutils import nodes
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
-from docutils.utils.error_reporting import ErrorString
 from sphinx.util.docutils import switch_source_input
 from sphinx.util import logging
 
-__version__  = '1.0'
+__version__ = "1.0"
+
 
 def setup(app):
 
     app.add_directive("kernel-abi", KernelCmd)
-    return dict(
-        version = __version__
-        , parallel_read_safe = True
-        , parallel_write_safe = True
-    )
+    return {
+        "version": __version__,
+        "parallel_read_safe": True,
+        "parallel_write_safe": True
+    }
+
 
 class KernelCmd(Directive):
-
     u"""KernelABI (``kernel-abi``) directive"""
 
     required_arguments = 1
@@ -92,15 +90,15 @@ class KernelCmd(Directive):
         nodeList = self.nestedParse(lines, self.arguments[0])
         return nodeList
 
-    def nestedParse(self, lines, fname):
+    def nested_parse(self, lines, fname):
         env = self.state.document.settings.env
         content = ViewList()
         node = nodes.section()
 
         if "debug" in self.options:
             code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
-            for l in lines.split("\n"):
-                code_block += "\n    " + l
+            for line in lines.split("\n"):
+                code_block += "\n    " + line
             lines = code_block + "\n\n"
 
         line_regex = re.compile(r"^\.\. LINENO (\S+)\#([0-9]+)$")
-- 
2.48.1

[RFC v2 06/38] docs: admin-guide: abi: add SPDX tags to ABI files

[Mauro Carvalho Chehab]

Such files are missing SPDX tags containing the licensing information.

Add them.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/admin-guide/abi-obsolete.rst | 2 ++
 Documentation/admin-guide/abi-removed.rst  | 2 ++
 Documentation/admin-guide/abi-stable.rst   | 2 ++
 Documentation/admin-guide/abi-testing.rst  | 2 ++
 Documentation/admin-guide/abi.rst          | 2 ++
 5 files changed, 10 insertions(+)

diff --git a/Documentation/admin-guide/abi-obsolete.rst b/Documentation/admin-guide/abi-obsolete.rst
index 594e697aa1b2..b655615917f1 100644
--- a/Documentation/admin-guide/abi-obsolete.rst
+++ b/Documentation/admin-guide/abi-obsolete.rst
@@ -1,3 +1,5 @@
+.. SPDX-License-Identifier: GPL-2.0
+
 ABI obsolete symbols
 ====================
 
diff --git a/Documentation/admin-guide/abi-removed.rst b/Documentation/admin-guide/abi-removed.rst
index f9e000c81828..ba941c1af178 100644
--- a/Documentation/admin-guide/abi-removed.rst
+++ b/Documentation/admin-guide/abi-removed.rst
@@ -1,3 +1,5 @@
+.. SPDX-License-Identifier: GPL-2.0
+
 ABI removed symbols
 ===================
 
diff --git a/Documentation/admin-guide/abi-stable.rst b/Documentation/admin-guide/abi-stable.rst
index fc3361d847b1..5d738f345333 100644
--- a/Documentation/admin-guide/abi-stable.rst
+++ b/Documentation/admin-guide/abi-stable.rst
@@ -1,3 +1,5 @@
+.. SPDX-License-Identifier: GPL-2.0
+
 ABI stable symbols
 ==================
 
diff --git a/Documentation/admin-guide/abi-testing.rst b/Documentation/admin-guide/abi-testing.rst
index 19767926b344..a867e6578bf7 100644
--- a/Documentation/admin-guide/abi-testing.rst
+++ b/Documentation/admin-guide/abi-testing.rst
@@ -1,3 +1,5 @@
+.. SPDX-License-Identifier: GPL-2.0
+
 ABI testing symbols
 ===================
 
diff --git a/Documentation/admin-guide/abi.rst b/Documentation/admin-guide/abi.rst
index bcab3ef2597c..64e772bde943 100644
--- a/Documentation/admin-guide/abi.rst
+++ b/Documentation/admin-guide/abi.rst
@@ -1,3 +1,5 @@
+.. SPDX-License-Identifier: GPL-2.0
+
 =====================
 Linux ABI description
 =====================
-- 
2.48.1

[RFC v2 07/38] ABI: sysfs-class-rfkill: fix kernelversion tags

[Mauro Carvalho Chehab]

Some kernelversion tags are missing colons. Add them to comply with
ABI description and produce right results when converted to html/pdf.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/ABI/removed/sysfs-class-rfkill |  2 +-
 Documentation/ABI/stable/sysfs-class-rfkill  | 12 ++++++------
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/Documentation/ABI/removed/sysfs-class-rfkill b/Documentation/ABI/removed/sysfs-class-rfkill
index f25174eafd55..20cb688af173 100644
--- a/Documentation/ABI/removed/sysfs-class-rfkill
+++ b/Documentation/ABI/removed/sysfs-class-rfkill
@@ -4,7 +4,7 @@ For details to this subsystem look at Documentation/driver-api/rfkill.rst.
 
 What:		/sys/class/rfkill/rfkill[0-9]+/claim
 Date:		09-Jul-2007
-KernelVersion	v2.6.22
+KernelVersion:	v2.6.22
 Contact:	linux-wireless@vger.kernel.org
 Description:	This file was deprecated because there no longer was a way to
 		claim just control over a single rfkill instance.
diff --git a/Documentation/ABI/stable/sysfs-class-rfkill b/Documentation/ABI/stable/sysfs-class-rfkill
index 037979f7dc4b..67b605e3dd16 100644
--- a/Documentation/ABI/stable/sysfs-class-rfkill
+++ b/Documentation/ABI/stable/sysfs-class-rfkill
@@ -16,7 +16,7 @@ Description: 	The rfkill class subsystem folder.
 
 What:		/sys/class/rfkill/rfkill[0-9]+/name
 Date:		09-Jul-2007
-KernelVersion	v2.6.22
+KernelVersion:	v2.6.22
 Contact:	linux-wireless@vger.kernel.org
 Description: 	Name assigned by driver to this key (interface or driver name).
 Values: 	arbitrary string.
@@ -24,7 +24,7 @@ Values: 	arbitrary string.
 
 What: 		/sys/class/rfkill/rfkill[0-9]+/type
 Date:		09-Jul-2007
-KernelVersion	v2.6.22
+KernelVersion:	v2.6.22
 Contact:	linux-wireless@vger.kernel.org
 Description: 	Driver type string ("wlan", "bluetooth", etc).
 Values: 	See include/linux/rfkill.h.
@@ -32,7 +32,7 @@ Values: 	See include/linux/rfkill.h.
 
 What:		/sys/class/rfkill/rfkill[0-9]+/persistent
 Date:		09-Jul-2007
-KernelVersion	v2.6.22
+KernelVersion:	v2.6.22
 Contact:	linux-wireless@vger.kernel.org
 Description: 	Whether the soft blocked state is initialised from non-volatile
 		storage at startup.
@@ -44,7 +44,7 @@ Values: 	A numeric value:
 
 What:		/sys/class/rfkill/rfkill[0-9]+/state
 Date:		09-Jul-2007
-KernelVersion	v2.6.22
+KernelVersion:	v2.6.22
 Contact:	linux-wireless@vger.kernel.org
 Description: 	Current state of the transmitter.
 		This file was scheduled to be removed in 2014, but due to its
@@ -67,7 +67,7 @@ Values: 	A numeric value.
 
 What:		/sys/class/rfkill/rfkill[0-9]+/hard
 Date:		12-March-2010
-KernelVersion	v2.6.34
+KernelVersion:	v2.6.34
 Contact:	linux-wireless@vger.kernel.org
 Description: 	Current hardblock state. This file is read only.
 Values: 	A numeric value.
@@ -81,7 +81,7 @@ Values: 	A numeric value.
 
 What:		/sys/class/rfkill/rfkill[0-9]+/soft
 Date:		12-March-2010
-KernelVersion	v2.6.34
+KernelVersion:	v2.6.34
 Contact:	linux-wireless@vger.kernel.org
 Description:	Current softblock state. This file is read and write.
 Values: 	A numeric value.
-- 
2.48.1

[RFC v2 08/38] ABI: sysfs-bus-coresight-*: fix kernelversion tags

[Mauro Carvalho Chehab]

Some kernelversion tags are missing colons. Add them to comply with
ABI description and produce right results when converted to html/pdf.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 .../testing/sysfs-bus-coresight-devices-cti   | 78 +++++++++----------
 .../testing/sysfs-bus-coresight-devices-tpdm  | 52 ++++++-------
 2 files changed, 65 insertions(+), 65 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti b/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti
index bf2869c413e7..a97b70f588da 100644
--- a/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti
+++ b/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti
@@ -1,241 +1,241 @@
 What:		/sys/bus/coresight/devices/<cti-name>/enable
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Enable/Disable the CTI hardware.
 
 What:		/sys/bus/coresight/devices/<cti-name>/powered
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Indicate if the CTI hardware is powered.
 
 What:		/sys/bus/coresight/devices/<cti-name>/ctmid
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Display the associated CTM ID
 
 What:		/sys/bus/coresight/devices/<cti-name>/nr_trigger_cons
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Number of devices connected to triggers on this CTI
 
 What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/name
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Name of connected device <N>
 
 What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/in_signals
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Input trigger signals from connected device <N>
 
 What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/in_types
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Functional types for the input trigger signals
 		from connected device <N>
 
 What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/out_signals
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Output trigger signals to connected device <N>
 
 What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/out_types
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Functional types for the output trigger signals
 		to connected device <N>
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/inout_sel
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Select the index for inen and outen registers.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/inen
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Read or write the CTIINEN register selected by inout_sel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/outen
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Read or write the CTIOUTEN register selected by inout_sel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/gate
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Read or write CTIGATE register.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/asicctl
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Read or write ASICCTL register.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/intack
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Write the INTACK register.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/appset
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Set CTIAPPSET register to activate channel. Read back to
 		determine current value of register.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/appclear
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Write APPCLEAR register to deactivate channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/apppulse
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Write APPPULSE to pulse a channel active for one clock
 		cycle.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/chinstatus
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Read current status of channel inputs.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/choutstatus
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) read current status of channel outputs.
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/triginstatus
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) read current status of input trigger signals
 
 What:		/sys/bus/coresight/devices/<cti-name>/regs/trigoutstatus
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) read current status of output trigger signals.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/trigin_attach
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Attach a CTI input trigger to a CTM channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/trigin_detach
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Detach a CTI input trigger from a CTM channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/trigout_attach
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Attach a CTI output trigger to a CTM channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/trigout_detach
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Detach a CTI output trigger from a CTM channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_gate_enable
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Enable CTIGATE for single channel (Write) or list enabled
 		channels through the gate (R).
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_gate_disable
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Disable CTIGATE for single channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_set
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Activate a single channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_clear
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Deactivate a single channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_pulse
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Pulse a single channel - activate for a single clock cycle.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/trigout_filtered
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) List of output triggers filtered across all connections.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/trig_filter_enable
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Enable or disable trigger output signal filtering.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_inuse
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) show channels with at least one attached trigger signal.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_free
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) show channels with no attached trigger signals.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_sel
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(RW) Write channel number to select a channel to view, read to
 		see selected channel number.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_in
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Read to see input triggers connected to selected view
 		channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_out
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Read) Read to see output triggers connected to selected view
 		channel.
 
 What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_reset
 Date:		March 2020
-KernelVersion	5.7
+KernelVersion:	5.7
 Contact:	Mike Leach or Mathieu Poirier
 Description:	(Write) Clear all channel / trigger programming.
diff --git a/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm b/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm
index bf710ea6e0ef..53cb454b60d0 100644
--- a/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm
+++ b/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm
@@ -1,6 +1,6 @@
 What:		/sys/bus/coresight/devices/<tpdm-name>/integration_test
 Date:		January 2023
-KernelVersion	6.2
+KernelVersion:	6.2
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(Write) Run integration test for tpdm. Integration test
@@ -14,7 +14,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/reset_dataset
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(Write) Reset the dataset of the tpdm.
@@ -24,7 +24,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_type
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the trigger type of the DSB for tpdm.
@@ -35,7 +35,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_ts
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the trigger timestamp of the DSB for tpdm.
@@ -46,7 +46,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_mode
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the programming mode of the DSB for tpdm.
@@ -60,7 +60,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/ctrl_idx
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the index number of the edge detection for the DSB
@@ -69,7 +69,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/ctrl_val
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		Write a data to control the edge detection corresponding to
@@ -85,7 +85,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/ctrl_mask
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		Write a data to mask the edge detection corresponding to the index
@@ -97,21 +97,21 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/edcr[0:15]
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		Read a set of the edge control value of the DSB in TPDM.
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/edcmr[0:7]
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		Read a set of the edge control mask of the DSB in TPDM.
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_patt/xpr[0:7]
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the value of the trigger pattern for the DSB
@@ -119,7 +119,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_patt/xpmr[0:7]
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the mask of the trigger pattern for the DSB
@@ -127,21 +127,21 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpr[0:7]
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the value of the pattern for the DSB subunit TPDM.
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpmr[0:7]
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the mask of the pattern for the DSB subunit TPDM.
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/enable_ts
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(Write) Set the pattern timestamp of DSB tpdm. Read
@@ -153,7 +153,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/set_type
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(Write) Set the pattern type of DSB tpdm. Read
@@ -165,7 +165,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_msr/msr[0:31]
 Date:		March 2023
-KernelVersion	6.7
+KernelVersion:	6.7
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the MSR(mux select register) for the DSB subunit
@@ -173,7 +173,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_mode
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:	(Write) Set the data collection mode of CMB tpdm. Continuous
 		change creates CMB data set elements on every CMBCLK edge.
@@ -187,7 +187,7 @@ Description:	(Write) Set the data collection mode of CMB tpdm. Continuous
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_trig_patt/xpr[0:1]
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the value of the trigger pattern for the CMB
@@ -195,7 +195,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_trig_patt/xpmr[0:1]
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the mask of the trigger pattern for the CMB
@@ -203,21 +203,21 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpr[0:1]
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the value of the pattern for the CMB subunit TPDM.
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpmr[0:1]
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the mask of the pattern for the CMB subunit TPDM.
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_patt/enable_ts
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(Write) Set the pattern timestamp of CMB tpdm. Read
@@ -229,7 +229,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_trig_ts
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the trigger timestamp of the CMB for tpdm.
@@ -240,7 +240,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_ts_all
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Read or write the status of timestamp upon all interface.
@@ -252,7 +252,7 @@ Description:
 
 What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_msr/msr[0:31]
 Date:		January 2024
-KernelVersion	6.9
+KernelVersion:	6.9
 Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
 Description:
 		(RW) Set/Get the MSR(mux select register) for the CMB subunit
-- 
2.48.1

[RFC v2 09/38] ABI: sysfs-driver-dma-idxd: fix date tags

[Mauro Carvalho Chehab]

Some date tags are missing colons. Add them to comply with
ABI description and produce right results when converted to html/pdf.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/ABI/stable/sysfs-driver-dma-idxd | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/stable/sysfs-driver-dma-idxd b/Documentation/ABI/stable/sysfs-driver-dma-idxd
index f2ec42949a54..4a355e6747ae 100644
--- a/Documentation/ABI/stable/sysfs-driver-dma-idxd
+++ b/Documentation/ABI/stable/sysfs-driver-dma-idxd
@@ -246,14 +246,14 @@ Description:	Controls whether PRS disable is turned on for the workqueue.
 		capability.
 
 What:		/sys/bus/dsa/devices/wq<m>.<n>/occupancy
-Date		May 25, 2021
+Date:		May 25, 2021
 KernelVersion:	5.14.0
 Contact:	dmaengine@vger.kernel.org
 Description:	Show the current number of entries in this WQ if WQ Occupancy
 		Support bit WQ capabilities is 1.
 
 What:		/sys/bus/dsa/devices/wq<m>.<n>/enqcmds_retries
-Date		Oct 29, 2021
+Date:		Oct 29, 2021
 KernelVersion:	5.17.0
 Contact:	dmaengine@vger.kernel.org
 Description:	Indicate the number of retires for an enqcmds submission on a sharedwq.
-- 
2.48.1

[RFC v2 10/38] ABI: sysfs-fs-f2fs: fix date tags

[Mauro Carvalho Chehab]

Some date tags are missing colons. Add them to comply with
ABI description and produce right results when converted to html/pdf.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/ABI/testing/sysfs-fs-f2fs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-fs-f2fs b/Documentation/ABI/testing/sysfs-fs-f2fs
index 3e1630c70d8a..e44bb614964b 100644
--- a/Documentation/ABI/testing/sysfs-fs-f2fs
+++ b/Documentation/ABI/testing/sysfs-fs-f2fs
@@ -347,7 +347,7 @@ Description:	Used to control configure extension list:
 		- [c] means add/del cold file extension
 
 What:		/sys/fs/f2fs/<disk>/unusable
-Date		April 2019
+Date:		April 2019
 Contact:	"Daniel Rosenberg" <drosen@google.com>
 Description:	If checkpoint=disable, it displays the number of blocks that
 		are unusable.
@@ -355,7 +355,7 @@ Description:	If checkpoint=disable, it displays the number of blocks that
 		would be unusable if checkpoint=disable were to be set.
 
 What:		/sys/fs/f2fs/<disk>/encoding
-Date		July 2019
+Date:		July 2019
 Contact:	"Daniel Rosenberg" <drosen@google.com>
 Description:	Displays name and version of the encoding set for the filesystem.
 		If no encoding is set, displays (none)
-- 
2.48.1

[RFC v2 11/38] ABI: sysfs-power: fix a what tag

[Mauro Carvalho Chehab]

There is one What tag that it is using semicolon instead of colon.

Fix it to comply with ABI description and produce right results when
converted to html/pdf.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/ABI/testing/sysfs-power | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/ABI/testing/sysfs-power b/Documentation/ABI/testing/sysfs-power
index a3942b1036e2..2192478e83cf 100644
--- a/Documentation/ABI/testing/sysfs-power
+++ b/Documentation/ABI/testing/sysfs-power
@@ -131,7 +131,7 @@ Description:
 		CAUTION: Using it will cause your machine's real-time (CMOS)
 		clock to be set to a random invalid time after a resume.
 
-What;		/sys/power/pm_trace_dev_match
+What:		/sys/power/pm_trace_dev_match
 Date:		October 2010
 Contact:	James Hogan <jhogan@kernel.org>
 Description:
-- 
2.48.1

[RFC v2 12/38] scripts/documentation-file-ref-check: don't check perl/python scripts

[Mauro Carvalho Chehab]

Such scripts may have regular expressions, which would make the
parser confusing. Also, they shouldn't hardcode filenames there,
so skipping them is OK.

While here, also don't check references on extensions used for file
backup and patch rej/orig.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/documentation-file-ref-check | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/documentation-file-ref-check b/scripts/documentation-file-ref-check
index 68083f2f1122..408b1dbe7884 100755
--- a/scripts/documentation-file-ref-check
+++ b/scripts/documentation-file-ref-check
@@ -92,7 +92,7 @@ while (<IN>) {
 	next if ($f =~ m,^Next/,);
 
 	# Makefiles and scripts contain nasty expressions to parse docs
-	next if ($f =~ m/Makefile/ || $f =~ m/\.sh$/);
+	next if ($f =~ m/Makefile/ || $f =~ m/\.(sh|py|pl|~|rej|org|orig)$/);
 
 	# It doesn't make sense to parse hidden files
 	next if ($f =~ m#/\.#);
-- 
2.48.1

[RFC v2 13/38] scripts/get_abi.py: add a Python tool to generate ReST output

[Mauro Carvalho Chehab]

The get_abi.pl script is requiring some care, but it seems that
the number of changes on it since when I originally wrote it
was not too high.

Maintaining perl scripts without using classes requires a higher
efforted than on python, due to global variables management.
Also, it sounds easier to find python developer those days than
perl ones.

As a plus, using a Python class to handle ABI allows a better
integration with Sphinx extensions, allowing, for instance,
to let automarkup to generate cross-references for ABI
symbols.

With that in mind, rewrite the core of get_abi.pl in Python,
using classes, to help producing documentation. This will
allow a better integration in the future with the Sphinx
ABI extension.

The algorithms used there are the same as the ones in Perl,
with a couple of cleanups to remove redundant variables and
to help with cross-reference generation. While doing that,
remove some code that were important in the past, where
ABI files weren't using ReST format.

Some minor improvements were added like using a fixed seed
when generating ABI keys for duplicated names, making its
results reproductible.

The end script is a little bit faster than the original one
(tested on a machine with ssd disks). That's probably because
we're now using only pre-compiled regular expressions, and it
is using string replacement methods instead of regex where
possible.

The new version is a little bit more conservative when
converting text to cross-references to avoid adding them into
literal blocks.

To ensure that the ReST output is parsing all variables
and files properly, the end result was compared using diff
with the one produced by the perl script and showed no regressions.
There are minor improvements at the results, as it now
properly groups What on some special cases. It also better
escape some XREF names.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 619 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 619 insertions(+)
 create mode 100755 scripts/get_abi.py

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
new file mode 100755
index 000000000000..0b5a10e76a3c
--- /dev/null
+++ b/scripts/get_abi.py
@@ -0,0 +1,619 @@
+#!/usr/bin/env python3
+# pylint: disable=R0902,R0903,R0911,R0912,R0913,R0914,R0915,R0917,C0302
+# Copyright(c) 2025: Mauro Carvalho Chehab <mchehab@kernel.org>.
+# SPDX-License-Identifier: GPL-2.0
+
+"""
+Parse ABI documentation and produce results from it.
+"""
+
+import argparse
+import logging
+import os
+import re
+import sys
+
+from glob import glob
+from pprint import pformat
+from random import randrange, seed
+
+ABI_DIR = "Documentation/ABI/"
+
+# Debug levels
+DEBUG_WHAT_PARSING = 1
+DEBUG_WHAT_OPEN = 2
+DEBUG_DUMP_ABI_STRUCTS = 4
+
+DEBUG_HELP = """
+Print debug information according with the level(s),
+which is given by the following bitmask:
+
+1  - enable debug parsing logic
+2  - enable debug messages on file open
+4  - enable debug for ABI parse data
+"""
+
+
+class AbiParser:
+    """Main class to parse ABI files"""
+
+    TAGS = r"(what|where|date|kernelversion|contact|description|users)"
+    XREF = r"(?:^|\s|\()(\/(?:sys|config|proc|dev|kvd)\/[^,.:;\)\s]+)(?:[,.:;\)\s]|\Z)"
+
+    def __init__(self, directory, logger=None,
+                 enable_lineno=False, show_warnings=True, debug=0):
+        """Stores arguments for the class and initialize class vars"""
+
+        self.directory = directory
+        self.enable_lineno = enable_lineno
+        self.show_warnings = show_warnings
+        self.debug = debug
+
+        if not logger:
+            self.log = logging.getLogger("get_abi")
+        else:
+            self.log = logger
+
+        self.data = {}
+        self.what_symbols = {}
+        self.file_refs = {}
+        self.what_refs = {}
+
+        # Regular expressions used on parser
+        self.re_tag = re.compile(r"(\S+)(:\s*)(.*)", re.I)
+        self.re_valid = re.compile(self.TAGS)
+        self.re_start_spc = re.compile(r"(\s*)(\S.*)")
+        self.re_whitespace = re.compile(r"^\s+")
+
+        # Regular used on print
+        self.re_what = re.compile(r"(\/?(?:[\w\-]+\/?){1,2})")
+        self.re_escape = re.compile(r"([\.\x01-\x08\x0e-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])")
+        self.re_unprintable = re.compile(r"([\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\xff]+)")
+        self.re_title_mark = re.compile(r"\n[\-\*\=\^\~]+\n")
+        self.re_doc = re.compile(r"Documentation/(?!devicetree)(\S+)\.rst")
+        self.re_abi = re.compile(r"(Documentation/ABI/)([\w\/\-]+)")
+        self.re_xref_node = re.compile(self.XREF)
+
+    def warn(self, fdata, msg, extra=None):
+        """Displays a parse error if warning is enabled"""
+
+        if not self.show_warnings:
+            return
+
+        msg = f"{fdata.fname}:{fdata.ln}: {msg}"
+        if extra:
+            msg += "\n\t\t" + extra
+
+        self.log.warning(msg)
+
+    def add_symbol(self, what, fname, ln=None, xref=None):
+        """Create a reference table describing where each 'what' is located"""
+
+        if what not in self.what_symbols:
+            self.what_symbols[what] = {"file": {}}
+
+        if fname not in self.what_symbols[what]["file"]:
+            self.what_symbols[what]["file"][fname] = []
+
+        if ln and ln not in self.what_symbols[what]["file"][fname]:
+            self.what_symbols[what]["file"][fname].append(ln)
+
+        if xref:
+            self.what_symbols[what]["xref"] = xref
+
+    def _parse_line(self, fdata, line):
+        """Parse a single line of an ABI file"""
+
+        new_what = False
+        new_tag = False
+        content = None
+
+        match = self.re_tag.match(line)
+        if match:
+            new = match.group(1).lower()
+            sep = match.group(2)
+            content = match.group(3)
+
+            match = self.re_valid.search(new)
+            if match:
+                new_tag = match.group(1)
+            else:
+                if fdata.tag == "description":
+                    # New "tag" is actually part of description.
+                    # Don't consider it a tag
+                    new_tag = False
+                elif fdata.tag != "":
+                    self.warn(fdata, f"tag '{fdata.tag}' is invalid", line)
+
+        if new_tag:
+            # "where" is Invalid, but was a common mistake. Warn if found
+            if new_tag == "where":
+                self.warn(fdata, "tag 'Where' is invalid. Should be 'What:' instead")
+                new_tag = "what"
+
+            if new_tag == "what":
+                fdata.space = None
+
+                if content not in self.what_symbols:
+                    self.add_symbol(what=content, fname=fdata.fname, ln=fdata.ln)
+
+                if fdata.tag == "what":
+                    fdata.what.append(content.strip("\n"))
+                else:
+                    if fdata.key:
+                        if "description" not in self.data.get(fdata.key, {}):
+                            self.warn(fdata, f"{fdata.key} doesn't have a description")
+
+                        for w in fdata.what:
+                            self.add_symbol(what=w, fname=fdata.fname,
+                                            ln=fdata.what_ln, xref=fdata.key)
+
+                    fdata.label = content
+                    new_what = True
+
+                    key = "abi_" + content.lower()
+                    fdata.key = self.re_unprintable.sub("_", key).strip("_")
+
+                    # Avoid duplicated keys but using a defined seed, to make
+                    # the namespace identical if there aren't changes at the
+                    # ABI symbols
+                    seed(42)
+
+                    while fdata.key in self.data:
+                        char = randrange(0, 51) + ord("A")
+                        if char > ord("Z"):
+                            char += ord("a") - ord("Z") - 1
+
+                        fdata.key += chr(char)
+
+                    if fdata.key and fdata.key not in self.data:
+                        self.data[fdata.key] = {
+                            "what": [content],
+                            "file": [fdata.file_ref],
+                            "line_no": fdata.ln,
+                        }
+
+                    fdata.what = self.data[fdata.key]["what"]
+
+                self.what_refs[content] = fdata.key
+                fdata.tag = new_tag
+                fdata.what_ln = fdata.ln
+
+                if fdata.nametag["what"]:
+                    t = (content, fdata.key)
+                    if t not in fdata.nametag["symbols"]:
+                        fdata.nametag["symbols"].append(t)
+
+                return
+
+            if fdata.tag and new_tag:
+                fdata.tag = new_tag
+
+                if new_what:
+                    fdata.label = ""
+
+                    self.data[fdata.key]["type"] = fdata.ftype
+
+                    if "description" in self.data[fdata.key]:
+                        self.data[fdata.key]["description"] += "\n\n"
+
+                    if fdata.file_ref not in self.data[fdata.key]["file"]:
+                        self.data[fdata.key]["file"].append(fdata.file_ref)
+
+                    if self.debug == DEBUG_WHAT_PARSING:
+                        self.log.debug("what: %s", fdata.what)
+
+                if not fdata.what:
+                    self.warn(fdata, "'What:' should come first:", line)
+                    return
+
+                if new_tag == "description":
+                    fdata.space = None
+
+                    if content:
+                        sep = sep.replace(":", " ")
+
+                        c = " " * len(new_tag) + sep + content
+                        c = c.expandtabs()
+
+                        match = self.re_start_spc.match(c)
+                        if match:
+                            # Preserve initial spaces for the first line
+                            fdata.space = match.group(1)
+                            content = match.group(2) + "\n"
+
+                self.data[fdata.key][fdata.tag] = content
+
+            return
+
+        # Store any contents before tags at the database
+        if not fdata.tag and "what" in fdata.nametag:
+            fdata.nametag["description"] += line
+            return
+
+        if fdata.tag == "description":
+            content = line.expandtabs()
+
+            if self.re_whitespace.sub("", content) == "":
+                self.data[fdata.key][fdata.tag] += "\n"
+                return
+
+            if fdata.space is None:
+                match = self.re_start_spc.match(content)
+                if match:
+                    # Preserve initial spaces for the first line
+                    fdata.space = match.group(1)
+
+                    content = match.group(2) + "\n"
+            else:
+                if content.startswith(fdata.space):
+                    content = content[len(fdata.space):]
+
+                else:
+                    fdata.space = ""
+
+            if fdata.tag == "what":
+                w = content.strip("\n")
+                if w:
+                    self.data[fdata.key][fdata.tag].append(w)
+            else:
+                self.data[fdata.key][fdata.tag] += content
+            return
+
+        content = line.strip()
+        if fdata.tag:
+            if fdata.tag == "what":
+                w = content.strip("\n")
+                if w:
+                    self.data[fdata.key][fdata.tag].append(w)
+            else:
+                self.data[fdata.key][fdata.tag] += "\n" + content.rstrip("\n")
+            return
+
+        # Everything else is error
+        if content:
+            self.warn(fdata, "Unexpected content", line)
+
+    def parse_file(self, fname, path, basename):
+        """Parse a single file"""
+
+        ref = f"abi_file_{path}_{basename}"
+        ref = self.re_unprintable.sub("_", ref).strip("_")
+
+        # Store per-file state into a namespace variable. This will be used
+        # by the per-line parser state machine and by the warning function.
+        fdata = argparse.Namespace
+
+        fdata.fname = fname
+        fdata.name = basename
+
+        pos = fname.find(ABI_DIR)
+        if pos > 0:
+            f = fname[pos:]
+        else:
+            f = fname
+
+        fdata.file_ref = (f, ref)
+        self.file_refs[f] = ref
+
+        fdata.ln = 0
+        fdata.what_ln = 0
+        fdata.tag = ""
+        fdata.label = ""
+        fdata.what = []
+        fdata.key = None
+        fdata.xrefs = None
+        fdata.space = None
+        fdata.ftype = path.split("/")[0]
+
+        fdata.nametag = {}
+        fdata.nametag["what"] = [f"File {path}/{basename}"]
+        fdata.nametag["type"] = "File"
+        fdata.nametag["file"] = [fdata.file_ref]
+        fdata.nametag["line_no"] = 1
+        fdata.nametag["description"] = ""
+        fdata.nametag["symbols"] = []
+
+        self.data[ref] = fdata.nametag
+
+        if self.debug & DEBUG_WHAT_OPEN:
+            self.log.debug("Opening file %s", fname)
+
+        with open(fname, "r", encoding="utf8", errors="backslashreplace") as fp:
+            for line in fp:
+                fdata.ln += 1
+
+                self._parse_line(fdata, line)
+
+            if "description" in fdata.nametag:
+                fdata.nametag["description"] = fdata.nametag["description"].lstrip("\n")
+
+            if fdata.key:
+                if "description" not in self.data.get(fdata.key, {}):
+                    self.warn(fdata, f"{fdata.key} doesn't have a description")
+
+                for w in fdata.what:
+                    self.add_symbol(what=w, fname=fname, xref=fdata.key)
+
+    def parse_abi(self):
+        """Parse documentation ABI"""
+
+        ignore_suffixes = ("rej", "org", "orig", "bak", "~")
+        re_abi = re.compile(r".*" + ABI_DIR)
+
+        for fname in glob(os.path.join(self.directory, "**"), recursive=True):
+            if os.path.isdir(fname):
+                continue
+
+            basename = os.path.basename(fname)
+
+            if basename == "README":
+                continue
+            if basename.startswith(".") or basename.endswith(ignore_suffixes):
+                continue
+
+            path = re_abi.sub("", os.path.dirname(fname))
+
+            self.parse_file(fname, path, basename)
+
+        if self.debug & DEBUG_DUMP_ABI_STRUCTS:
+            self.log.debug(pformat(self.data))
+
+    def print_desc_txt(self, desc):
+        """Print description as found inside ABI files"""
+
+        desc = desc.strip(" \t\n")
+
+        print(desc + "\n")
+
+    def print_desc_rst(self, desc):
+        """Enrich ReST output by creating cross-references"""
+
+        # Remove title markups from the description
+        # Having titles inside ABI files will only work if extra
+        # care would be taken in order to strictly follow the same
+        # level order for each markup.
+        desc = self.re_title_mark.sub("\n\n", "\n" + desc)
+        desc = desc.rstrip(" \t\n").lstrip("\n")
+
+
+        # FIXME: Python's regex performance for non-compiled expressions
+        # is a lot worse than Perl, as Perl automatically caches them at their
+        # first usage. Here, we'll need to do the same, as otherwise the
+        # performance penalty is be high
+
+        new_desc = ""
+        for d in desc.split("\n"):
+            if d == "":
+                new_desc += "\n"
+                continue
+
+            # Use cross-references for doc files where needed
+            d = self.re_doc.sub(r":doc:`/\1`", d)
+
+            # Use cross-references for ABI generated docs where needed
+            matches = self.re_abi.findall(d)
+            for m in matches:
+                abi = m[0] + m[1]
+
+                xref = self.file_refs.get(abi)
+                if not xref:
+                    # This may happen if ABI is on a separate directory,
+                    # like parsing ABI testing and symbol is at stable.
+                    # The proper solution is to move this part of the code
+                    # for it to be inside sphinx/kernel_abi.py
+                    self.log.info("Didn't find ABI reference for '%s'", abi)
+                else:
+                    new = self.re_escape.sub(r"\\\1", m[1])
+                    d = re.sub(fr"\b{abi}\b", f":ref:`{new} <{xref}>`", d)
+
+            # Seek for cross reference symbols like /sys/...
+            # Need to be careful to avoid doing it on a code block
+            if d[0] not in [" ", "\t"]:
+                matches = self.re_xref_node.findall(d)
+                for m in matches:
+                    # Finding ABI here is more complex due to wildcards
+                    xref = self.what_refs.get(m)
+                    if xref:
+                        new = self.re_escape.sub(r"\\\1", m)
+                        d = re.sub(fr"\b{m}\b", f":ref:`{new} <{xref}>`", d)
+
+            new_desc += d + "\n"
+
+        print(new_desc + "\n")
+
+    def print_data(self, enable_lineno, output_in_txt, show_file=False):
+        """Print ABI at stdout"""
+
+        part = None
+        for key, v in sorted(self.data.items(),
+                             key=lambda x: (x[1].get("type", ""),
+                                            x[1].get("what"))):
+
+            wtype = v.get("type", "Var")
+            file_ref = v.get("file")
+            names = v.get("what", [""])
+
+            if not show_file and wtype == "File":
+                continue
+
+            if enable_lineno:
+                ln = v.get("line_no", 1)
+                print(f".. LINENO {file_ref[0][0]}#{ln}\n")
+
+            if wtype != "File":
+                cur_part = names[0]
+                if cur_part.find("/") >= 0:
+                    match = self.re_what.match(cur_part)
+                    if match:
+                        symbol = match.group(1).rstrip("/")
+                        cur_part = "Symbols under " + symbol
+
+                if cur_part and cur_part != part:
+                    part = cur_part
+                    print(f"{part}\n{"-" * len(part)}\n")
+
+                print(f".. _{key}:\n")
+
+                max_len = 0
+                for i in range(0, len(names)):           # pylint: disable=C0200
+                    names[i] = "**" + self.re_escape.sub(r"\\\1", names[i]) + "**"
+
+                    max_len = max(max_len, len(names[i]))
+
+                print("+-" + "-" * max_len + "-+")
+                for name in names:
+                    print(f"| {name}" + " " * (max_len - len(name)) + " |")
+                    print("+-" + "-" * max_len + "-+")
+                print()
+
+            for ref in file_ref:
+                if wtype == "File":
+                    print(f".. _{ref[1]}:\n")
+                else:
+                    base = os.path.basename(ref[0])
+                    print(f"Defined on file :ref:`{base} <{ref[1]}>`\n")
+
+            if wtype == "File":
+                print(f"{names[0]}\n{"-" * len(names[0])}\n")
+
+            desc = v.get("description")
+            if not desc and wtype != "File":
+                print(f"DESCRIPTION MISSING for {names[0]}\n")
+
+            if desc:
+                if output_in_txt:
+                    self.print_desc_txt(desc)
+                else:
+                    self.print_desc_rst(desc)
+
+            symbols = v.get("symbols")
+            if symbols:
+                print("Has the following ABI:\n")
+
+                for w, label in symbols:
+                    # Escape special chars from content
+                    content = self.re_escape.sub(r"\\\1", w)
+
+                    print(f"- :ref:`{content} <{label}>`\n")
+
+            users = v.get("users")
+            if users and users.strip(" \t\n"):
+                print(f"Users:\n\t{users.strip("\n").replace('\n', '\n\t')}\n")
+
+    def check_issues(self):
+        """Warn about duplicated ABI entries"""
+
+        for what, v in self.what_symbols.items():
+            files = v.get("file")
+            if not files:
+                # Should never happen if the parser works properly
+                self.log.warning("%s doesn't have a file associated", what)
+                continue
+
+            if len(files) == 1:
+                continue
+
+            f = []
+            for fname, lines in sorted(files.items()):
+                if not lines:
+                    f.append(f"{fname}")
+                elif len(lines) == 1:
+                    f.append(f"{fname}:{lines[0]}")
+                else:
+                    f.append(f"{fname} lines {", ".join(str(x) for x in lines)}")
+
+            self.log.warning("%s is defined %d times: %s", what, len(f), "; ".join(f))
+
+
+class AbiRest:
+    """Initialize an argparse subparser for rest output"""
+
+    def __init__(self, subparsers):
+        """Initialize argparse subparsers"""
+
+        parser = subparsers.add_parser("rest",
+                                       formatter_class=argparse.RawTextHelpFormatter,
+                                       description=REST_DESC)
+
+        parser.add_argument("--enable-lineno",  action="store_true",
+                            help="enable lineno")
+        parser.add_argument("--raw", action="store_true",
+                            help="output text as contained in the ABI files. "
+                                 "It not used, output will contain dynamically"
+                                 " generated cross references when possible.")
+        parser.add_argument("--no-file", action="store_true",
+                            help="Don't the files section")
+        parser.add_argument("--show-hints", help="Show-hints")
+
+        parser.set_defaults(func=self.run)
+
+    def run(self, args):
+        """Run subparser"""
+
+        parser = AbiParser(args.dir, debug=args.debug)
+        parser.parse_abi()
+        parser.check_issues()
+        parser.print_data(args.enable_lineno, args.raw, not args.no_file)
+
+
+class AbiValidate:
+    """Initialize an argparse subparser for ABI validation"""
+
+    def __init__(self, subparsers):
+        """Initialize argparse subparsers"""
+
+        parser = subparsers.add_parser("validate",
+                                       formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+                                       description="list events")
+
+        parser.set_defaults(func=self.run)
+
+    def run(self, args):
+        """Run subparser"""
+
+        parser = AbiParser(args.dir, debug=args.debug)
+        parser.parse_abi()
+        parser.check_issues()
+
+
+def main():
+    """Main program"""
+
+    parser = argparse.ArgumentParser(formatter_class=argparse.RawTextHelpFormatter)
+
+    parser.add_argument("-d", "--debug", type=int, default=0, help="debug level")
+    parser.add_argument("-D", "--dir", default=ABI_DIR, help=DEBUG_HELP)
+
+    subparsers = parser.add_subparsers()
+
+    AbiRest(subparsers)
+    AbiValidate(subparsers)
+
+    args = parser.parse_args()
+
+    if args.debug:
+        level = logging.DEBUG
+    else:
+        level = logging.INFO
+
+    logging.basicConfig(level=level, format="[%(levelname)s] %(message)s")
+
+    if "func" in args:
+        args.func(args)
+    else:
+        sys.exit(f"Please specify a valid command for {sys.argv[0]}")
+
+
+REST_DESC = """
+Produce output in ReST format.
+
+The output is done on two sections:
+
+- Symbols: show all parsed symbols in alphabetic order;
+- Files: cross reference the content of each file with the symbols on it.
+"""
+
+# Call main method
+if __name__ == "__main__":
+    main()
-- 
2.48.1

[RFC v2 14/38] scripts/get_abi.py: add support for symbol search

[Mauro Carvalho Chehab]

Add support for searching symbols from Documentation/ABI using
regular expressions to match the symbols' names.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 76 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 0b5a10e76a3c..93a23cf30ea1 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -525,6 +525,58 @@ class AbiParser:
 
             self.log.warning("%s is defined %d times: %s", what, len(f), "; ".join(f))
 
+    def search_symbols(self, expr):
+        """ Searches for ABI symbols """
+
+        regex = re.compile(expr, re.I)
+
+        found_keys = 0
+        for t in sorted(self.data.items(), key=lambda x: [0]):
+            v = t[1]
+
+            wtype = v.get("type", "")
+            if wtype == "File":
+                continue
+
+            for what in v.get("what", [""]):
+                if regex.search(what):
+                    found_keys += 1
+
+                    kernelversion = v.get("kernelversion", "").strip(" \t\n")
+                    date = v.get("date", "").strip(" \t\n")
+                    contact = v.get("contact", "").strip(" \t\n")
+                    users = v.get("users", "").strip(" \t\n")
+                    desc = v.get("description", "").strip(" \t\n")
+
+                    files = []
+                    for f in v.get("file", ()):
+                        files.append(f[0])
+
+                    what = str(found_keys) + ". " + what
+                    title_tag = "-" * len(what)
+
+                    print(f"\n{what}\n{title_tag}\n")
+
+                    if kernelversion:
+                        print(f"Kernel version:\t\t{kernelversion}")
+
+                    if date:
+                        print(f"Date:\t\t\t{date}")
+
+                    if contact:
+                        print(f"Contact:\t\t{contact}")
+
+                    if users:
+                        print(f"Users:\t\t\t{users}")
+
+                    print(f"Defined on file{'s'[:len(files) ^ 1]}:\t{", ".join(files)}")
+
+                    if desc:
+                        print(f"\n{desc.strip("\n")}\n")
+
+        if not found_keys:
+            print(f"Regular expression /{expr}/ not found.")
+
 
 class AbiRest:
     """Initialize an argparse subparser for rest output"""
@@ -577,6 +629,29 @@ class AbiValidate:
         parser.check_issues()
 
 
+class AbiSearch:
+    """Initialize an argparse subparser for ABI search"""
+
+    def __init__(self, subparsers):
+        """Initialize argparse subparsers"""
+
+        parser = subparsers.add_parser("search",
+                                       formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+                                       description="Search ABI using a regular expression")
+
+        parser.add_argument("expression",
+                            help="Case-insensitive search pattern for the ABI symbol")
+
+        parser.set_defaults(func=self.run)
+
+    def run(self, args):
+        """Run subparser"""
+
+        parser = AbiParser(args.dir, debug=args.debug)
+        parser.parse_abi()
+        parser.search_symbols(args.expression)
+
+
 def main():
     """Main program"""
 
@@ -589,6 +664,7 @@ def main():
 
     AbiRest(subparsers)
     AbiValidate(subparsers)
+    AbiSearch(subparsers)
 
     args = parser.parse_args()
 
-- 
2.48.1

[RFC v2 15/38] docs: use get_abi.py for ABI generation

[Mauro Carvalho Chehab]

Use the new script instead of the old one when generating ABI docs.

For now, execute it via exec. Future changes will better integrate it
by using the class defined there directly.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/admin-guide/abi-obsolete.rst |  1 -
 Documentation/admin-guide/abi-removed.rst  |  1 -
 Documentation/admin-guide/abi-stable.rst   |  1 -
 Documentation/admin-guide/abi-testing.rst  |  1 -
 Documentation/sphinx/kernel_abi.py         | 10 +++-------
 5 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/Documentation/admin-guide/abi-obsolete.rst b/Documentation/admin-guide/abi-obsolete.rst
index b655615917f1..6d4d9ab7b8c3 100644
--- a/Documentation/admin-guide/abi-obsolete.rst
+++ b/Documentation/admin-guide/abi-obsolete.rst
@@ -10,4 +10,3 @@ The description of the interface will document the reason why it is
 obsolete and when it can be expected to be removed.
 
 .. kernel-abi:: ABI/obsolete
-   :rst:
diff --git a/Documentation/admin-guide/abi-removed.rst b/Documentation/admin-guide/abi-removed.rst
index ba941c1af178..9fc78af6f077 100644
--- a/Documentation/admin-guide/abi-removed.rst
+++ b/Documentation/admin-guide/abi-removed.rst
@@ -4,4 +4,3 @@ ABI removed symbols
 ===================
 
 .. kernel-abi:: ABI/removed
-   :rst:
diff --git a/Documentation/admin-guide/abi-stable.rst b/Documentation/admin-guide/abi-stable.rst
index 5d738f345333..c47c2a295865 100644
--- a/Documentation/admin-guide/abi-stable.rst
+++ b/Documentation/admin-guide/abi-stable.rst
@@ -13,4 +13,3 @@ Most interfaces (like syscalls) are expected to never change and always
 be available.
 
 .. kernel-abi:: ABI/stable
-   :rst:
diff --git a/Documentation/admin-guide/abi-testing.rst b/Documentation/admin-guide/abi-testing.rst
index a867e6578bf7..40b31985e587 100644
--- a/Documentation/admin-guide/abi-testing.rst
+++ b/Documentation/admin-guide/abi-testing.rst
@@ -19,4 +19,3 @@ name to the description of these interfaces, so that the kernel
 developers can easily notify them if any changes occur.
 
 .. kernel-abi:: ABI/testing
-   :rst:
diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index a1887b50a900..fc7500fad119 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -14,7 +14,7 @@ u"""
     :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
 
     The ``kernel-abi`` (:py:class:`KernelCmd`) directive calls the
-    scripts/get_abi.pl script to parse the Kernel ABI files.
+    scripts/get_abi.py script to parse the Kernel ABI files.
 
     Overview of directive's argument and options.
 
@@ -66,7 +66,6 @@ class KernelCmd(Directive):
 
     option_spec = {
         "debug"     : directives.flag,
-        "rst"       : directives.unchanged
     }
 
     def run(self):
@@ -77,15 +76,12 @@ class KernelCmd(Directive):
         srctree = os.path.abspath(os.environ["srctree"])
 
         args = [
-            os.path.join(srctree, 'scripts/get_abi.pl'),
+            os.path.join(srctree, 'scripts/get_abi.py'),
+            '-D', os.path.join(srctree, 'Documentation', self.arguments[0]),
             'rest',
             '--enable-lineno',
-            '--dir', os.path.join(srctree, 'Documentation', self.arguments[0]),
         ]
 
-        if 'rst' in self.options:
-            args.append('--rst-source')
-
         lines = subprocess.check_output(args, cwd=os.path.dirname(doc.current_source)).decode('utf-8')
         nodeList = self.nestedParse(lines, self.arguments[0])
         return nodeList
-- 
2.48.1

[RFC v2 16/38] scripts/get_abi.py: optimize parse_abi() function

[Mauro Carvalho Chehab]

Instead of using glob, use a recursive function to parse all files.

Such change reduces the total excecution time by 15% with my SSD disks.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 57 ++++++++++++++++++++++++++++++----------------
 1 file changed, 38 insertions(+), 19 deletions(-)

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 93a23cf30ea1..73613fb29c1c 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -13,7 +13,6 @@ import os
 import re
 import sys
 
-from glob import glob
 from pprint import pformat
 from random import randrange, seed
 
@@ -59,7 +58,11 @@ class AbiParser:
         self.file_refs = {}
         self.what_refs = {}
 
+        # Ignore files that contain such suffixes
+        self.ignore_suffixes = (".rej", ".org", ".orig", ".bak", "~")
+
         # Regular expressions used on parser
+        self.re_abi_dir = re.compile(r"(.*)" + ABI_DIR)
         self.re_tag = re.compile(r"(\S+)(:\s*)(.*)", re.I)
         self.re_valid = re.compile(self.TAGS)
         self.re_start_spc = re.compile(r"(\s*)(\S.*)")
@@ -335,26 +338,42 @@ class AbiParser:
                 for w in fdata.what:
                     self.add_symbol(what=w, fname=fname, xref=fdata.key)
 
-    def parse_abi(self):
+    def _parse_abi(self, root=None):
+        """Internal function to parse documentation ABI recursively"""
+
+        if not root:
+            root = self.directory
+
+        with os.scandir(root) as obj:
+            for entry in obj:
+                name = os.path.join(root, entry.name)
+
+                if entry.is_dir():
+                    self.parse_abi(name)
+                    continue
+
+                if not entry.is_file():
+                    continue
+
+                basename = os.path.basename(name)
+
+                if basename == "README":
+                    continue
+
+                if basename.startswith("."):
+                    continue
+
+                if basename.endswith(self.ignore_suffixes):
+                    continue
+
+                path = self.re_abi_dir.sub("", os.path.dirname(name))
+
+                self.parse_file(name, path, basename)
+
+    def parse_abi(self, root=None):
         """Parse documentation ABI"""
 
-        ignore_suffixes = ("rej", "org", "orig", "bak", "~")
-        re_abi = re.compile(r".*" + ABI_DIR)
-
-        for fname in glob(os.path.join(self.directory, "**"), recursive=True):
-            if os.path.isdir(fname):
-                continue
-
-            basename = os.path.basename(fname)
-
-            if basename == "README":
-                continue
-            if basename.startswith(".") or basename.endswith(ignore_suffixes):
-                continue
-
-            path = re_abi.sub("", os.path.dirname(fname))
-
-            self.parse_file(fname, path, basename)
+        self._parse_abi(root)
 
         if self.debug & DEBUG_DUMP_ABI_STRUCTS:
             self.log.debug(pformat(self.data))
-- 
2.48.1

[RFC v2 17/38] scripts/get_abi.py: use an interactor for ReST output

[Mauro Carvalho Chehab]

Instead of printing all results line per line, use an interactor
to return each variable as a separate message.

This won't change much when using it via command line, but it
will help Sphinx integration by providing an interactor that
could be used there to handle ABI symbol by symbol.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 52 ++++++++++++++++++++++++++--------------------
 1 file changed, 29 insertions(+), 23 deletions(-)

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 73613fb29c1c..2aec1f9dc5aa 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -349,7 +349,7 @@ class AbiParser:
                 name = os.path.join(root, entry.name)
 
                 if entry.is_dir():
-                    self.parse_abi(name)
+                    self._parse_abi(name)
                     continue
 
                 if not entry.is_file():
@@ -378,14 +378,14 @@ class AbiParser:
         if self.debug & DEBUG_DUMP_ABI_STRUCTS:
             self.log.debug(pformat(self.data))
 
-    def print_desc_txt(self, desc):
+    def desc_txt(self, desc):
         """Print description as found inside ABI files"""
 
         desc = desc.strip(" \t\n")
 
-        print(desc + "\n")
+        return desc + "\n\n"
 
-    def print_desc_rst(self, desc):
+    def desc_rst(self, desc):
         """Enrich ReST output by creating cross-references"""
 
         # Remove title markups from the description
@@ -439,9 +439,9 @@ class AbiParser:
 
             new_desc += d + "\n"
 
-        print(new_desc + "\n")
+        return new_desc + "\n\n"
 
-    def print_data(self, enable_lineno, output_in_txt, show_file=False):
+    def doc(self, enable_lineno, output_in_txt, show_file=False):
         """Print ABI at stdout"""
 
         part = None
@@ -456,9 +456,11 @@ class AbiParser:
             if not show_file and wtype == "File":
                 continue
 
+            msg = ""
+
             if enable_lineno:
                 ln = v.get("line_no", 1)
-                print(f".. LINENO {file_ref[0][0]}#{ln}\n")
+                msg += f".. LINENO {file_ref[0][0]}#{ln}\n\n"
 
             if wtype != "File":
                 cur_part = names[0]
@@ -470,9 +472,9 @@ class AbiParser:
 
                 if cur_part and cur_part != part:
                     part = cur_part
-                    print(f"{part}\n{"-" * len(part)}\n")
+                    msg += f"{part}\n{"-" * len(part)}\n\n"
 
-                print(f".. _{key}:\n")
+                msg += f".. _{key}:\n\n"
 
                 max_len = 0
                 for i in range(0, len(names)):           # pylint: disable=C0200
@@ -480,45 +482,47 @@ class AbiParser:
 
                     max_len = max(max_len, len(names[i]))
 
-                print("+-" + "-" * max_len + "-+")
+                msg += "+-" + "-" * max_len + "-+\n"
                 for name in names:
-                    print(f"| {name}" + " " * (max_len - len(name)) + " |")
-                    print("+-" + "-" * max_len + "-+")
-                print()
+                    msg += f"| {name}" + " " * (max_len - len(name)) + " |\n"
+                    msg += "+-" + "-" * max_len + "-+\n"
+                msg += "\n"
 
             for ref in file_ref:
                 if wtype == "File":
-                    print(f".. _{ref[1]}:\n")
+                    msg += f".. _{ref[1]}:\n\n"
                 else:
                     base = os.path.basename(ref[0])
-                    print(f"Defined on file :ref:`{base} <{ref[1]}>`\n")
+                    msg += f"Defined on file :ref:`{base} <{ref[1]}>`\n\n"
 
             if wtype == "File":
-                print(f"{names[0]}\n{"-" * len(names[0])}\n")
+                msg += f"{names[0]}\n{"-" * len(names[0])}\n\n"
 
             desc = v.get("description")
             if not desc and wtype != "File":
-                print(f"DESCRIPTION MISSING for {names[0]}\n")
+                msg += f"DESCRIPTION MISSING for {names[0]}\n\n"
 
             if desc:
                 if output_in_txt:
-                    self.print_desc_txt(desc)
+                    msg += self.desc_txt(desc)
                 else:
-                    self.print_desc_rst(desc)
+                    msg += self.desc_rst(desc)
 
             symbols = v.get("symbols")
             if symbols:
-                print("Has the following ABI:\n")
+                msg += "Has the following ABI:\n\n"
 
                 for w, label in symbols:
                     # Escape special chars from content
                     content = self.re_escape.sub(r"\\\1", w)
 
-                    print(f"- :ref:`{content} <{label}>`\n")
+                    msg += f"- :ref:`{content} <{label}>`\n\n"
 
             users = v.get("users")
             if users and users.strip(" \t\n"):
-                print(f"Users:\n\t{users.strip("\n").replace('\n', '\n\t')}\n")
+                msg += f"Users:\n\t{users.strip("\n").replace('\n', '\n\t')}\n\n"
+
+            yield msg
 
     def check_issues(self):
         """Warn about duplicated ABI entries"""
@@ -625,7 +629,9 @@ class AbiRest:
         parser = AbiParser(args.dir, debug=args.debug)
         parser.parse_abi()
         parser.check_issues()
-        parser.print_data(args.enable_lineno, args.raw, not args.no_file)
+
+        for msg in parser.doc(args.enable_lineno, args.raw, not args.no_file):
+            print(msg)
 
 
 class AbiValidate:
-- 
2.48.1

[RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Mauro Carvalho Chehab]

Instead of running get_abi.py script, import AbiParser class and
handle messages directly there using an interactor. This shold save some
memory, as there's no need to exec python inside the Sphinx python
extension.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/kernel_abi.py | 26 +++++++++++++++-----------
 scripts/get_abi.py                 |  2 +-
 2 files changed, 16 insertions(+), 12 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index fc7500fad119..93d537d8cb6c 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -42,6 +42,11 @@ from docutils.parsers.rst import directives, Directive
 from sphinx.util.docutils import switch_source_input
 from sphinx.util import logging
 
+srctree = os.path.abspath(os.environ["srctree"])
+sys.path.insert(0, os.path.join(srctree, "scripts"))
+
+from get_abi import AbiParser
+
 __version__ = "1.0"
 
 
@@ -65,7 +70,7 @@ class KernelCmd(Directive):
     logger = logging.getLogger('kernel_abi')
 
     option_spec = {
-        "debug"     : directives.flag,
+        "debug": directives.flag,
     }
 
     def run(self):
@@ -73,18 +78,17 @@ class KernelCmd(Directive):
         if not doc.settings.file_insertion_enabled:
             raise self.warning("docutils: file insertion disabled")
 
-        srctree = os.path.abspath(os.environ["srctree"])
+        path = os.path.join(srctree, "Documentation", self.arguments[0])
+        parser = AbiParser(path, logger=self.logger)
+        parser.parse_abi()
+        parser.check_issues()
 
-        args = [
-            os.path.join(srctree, 'scripts/get_abi.py'),
-            '-D', os.path.join(srctree, 'Documentation', self.arguments[0]),
-            'rest',
-            '--enable-lineno',
-        ]
+        msg = ""
+        for m in parser.doc(enable_lineno=True, show_file=True):
+            msg += m
 
-        lines = subprocess.check_output(args, cwd=os.path.dirname(doc.current_source)).decode('utf-8')
-        nodeList = self.nestedParse(lines, self.arguments[0])
-        return nodeList
+        node = self.nested_parse(msg, self.arguments[0])
+        return node
 
     def nested_parse(self, lines, fname):
         env = self.state.document.settings.env
diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 2aec1f9dc5aa..3a8dcff85dc2 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -441,7 +441,7 @@ class AbiParser:
 
         return new_desc + "\n\n"
 
-    def doc(self, enable_lineno, output_in_txt, show_file=False):
+    def doc(self, enable_lineno=False, output_in_txt=False, show_file=False):
         """Print ABI at stdout"""
 
         part = None
-- 
2.48.1

[RFC v2 19/38] docs: sphinx/kernel_abi: reduce buffer usage for ABI messages

[Mauro Carvalho Chehab]

Instead of producing a big message with all ABI contents and then
parse as a whole, simplify the code by handling each ABI symbol
in separate. As an additional benefit, there's no need to place
file/line nubers inlined at the data and use a regex to convert
them.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/kernel_abi.py | 76 +++++++++++++++---------------
 scripts/get_abi.py                 | 17 +++----
 2 files changed, 48 insertions(+), 45 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 93d537d8cb6c..025a4f02e1a4 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -68,6 +68,7 @@ class KernelCmd(Directive):
     has_content = False
     final_argument_whitespace = True
     logger = logging.getLogger('kernel_abi')
+    parser = None
 
     option_spec = {
         "debug": directives.flag,
@@ -79,59 +80,60 @@ class KernelCmd(Directive):
             raise self.warning("docutils: file insertion disabled")
 
         path = os.path.join(srctree, "Documentation", self.arguments[0])
-        parser = AbiParser(path, logger=self.logger)
-        parser.parse_abi()
-        parser.check_issues()
+        self.parser = AbiParser(path, logger=self.logger)
+        self.parser.parse_abi()
+        self.parser.check_issues()
 
-        msg = ""
-        for m in parser.doc(enable_lineno=True, show_file=True):
-            msg += m
-
-        node = self.nested_parse(msg, self.arguments[0])
+        node = self.nested_parse(None, self.arguments[0])
         return node
 
-    def nested_parse(self, lines, fname):
+    def nested_parse(self, data, fname):
         env = self.state.document.settings.env
         content = ViewList()
         node = nodes.section()
 
-        if "debug" in self.options:
-            code_block = "\n\n.. code-block:: rst\n    :linenos:\n"
-            for line in lines.split("\n"):
-                code_block += "\n    " + line
-            lines = code_block + "\n\n"
+        if data is not None:
+            # Handles the .rst file
+            for line in data.split("\n"):
+                content.append(line, fname, 0)
 
-        line_regex = re.compile(r"^\.\. LINENO (\S+)\#([0-9]+)$")
-        ln = 0
-        n = 0
-        f = fname
+            self.do_parse(content, node)
 
-        for line in lines.split("\n"):
-            n = n + 1
-            match = line_regex.search(line)
-            if match:
-                new_f = match.group(1)
+        else:
+            # Handles the ABI parser content, symbol by symbol
 
-                # Sphinx parser is lazy: it stops parsing contents in the
-                # middle, if it is too big. So, handle it per input file
-                if new_f != f and content:
-                    self.do_parse(content, node)
-                    content = ViewList()
+            old_f = fname
+            n = 0
+            for msg, f, ln in self.parser.doc():
+                msg_list = msg.split("\n")
+                if "debug" in self.options:
+                    lines = [
+                        "", "",  ".. code-block:: rst",
+                        "    :linenos:", ""
+                    ]
+                    for m in msg_list:
+                        lines.append("    " + m)
+                else:
+                    lines = msg_list
 
+                for line in lines:
+                    # sphinx counts lines from 0
+                    content.append(line, f, ln - 1)
+                    n += 1
+
+                if f != old_f:
                     # Add the file to Sphinx build dependencies
                     env.note_dependency(os.path.abspath(f))
 
-                f = new_f
+                    old_f = f
 
-                # sphinx counts lines from 0
-                ln = int(match.group(2)) - 1
-            else:
-                content.append(line, f, ln)
+                # Sphinx doesn't like to parse big messages. So, let's
+                # add content symbol by symbol
+                if content:
+                    self.do_parse(content, node)
+                    content = ViewList()
 
-        self.logger.info("%s: parsed %i lines" % (fname, n))
-
-        if content:
-            self.do_parse(content, node)
+            self.logger.info("%s: parsed %i lines" % (fname, n))
 
         return node.children
 
diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 3a8dcff85dc2..b57f46b91828 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -441,7 +441,7 @@ class AbiParser:
 
         return new_desc + "\n\n"
 
-    def doc(self, enable_lineno=False, output_in_txt=False, show_file=False):
+    def doc(self, output_in_txt=False, show_file=True):
         """Print ABI at stdout"""
 
         part = None
@@ -458,10 +458,6 @@ class AbiParser:
 
             msg = ""
 
-            if enable_lineno:
-                ln = v.get("line_no", 1)
-                msg += f".. LINENO {file_ref[0][0]}#{ln}\n\n"
-
             if wtype != "File":
                 cur_part = names[0]
                 if cur_part.find("/") >= 0:
@@ -522,7 +518,9 @@ class AbiParser:
             if users and users.strip(" \t\n"):
                 msg += f"Users:\n\t{users.strip("\n").replace('\n', '\n\t')}\n\n"
 
-            yield msg
+            ln = v.get("line_no", 1)
+
+            yield (msg, file_ref[0][0], ln)
 
     def check_issues(self):
         """Warn about duplicated ABI entries"""
@@ -630,8 +628,11 @@ class AbiRest:
         parser.parse_abi()
         parser.check_issues()
 
-        for msg in parser.doc(args.enable_lineno, args.raw, not args.no_file):
-            print(msg)
+        for t in parser.doc(args.raw, not args.no_file):
+            if args.enable_lineno:
+                print (f".. LINENO {t[1]}#{t[2]}\n\n")
+
+            print(t[0])
 
 
 class AbiValidate:
-- 
2.48.1

[RFC v2 20/38] docs: sphinx/kernel_abi: properly split lines

[Mauro Carvalho Chehab]

Sphinx doesn't like to have lines split with str.split("\n").

Instead, it uses its own splitter, with handles line breaks the way
Spinx expects. Not using it cause issues at the output files.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/kernel_abi.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 025a4f02e1a4..2408ba7a3981 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -36,7 +36,7 @@ import os
 import re
 import sys
 
-from docutils import nodes
+from docutils import nodes, statemachine
 from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives, Directive
 from sphinx.util.docutils import switch_source_input
@@ -105,7 +105,8 @@ class KernelCmd(Directive):
             old_f = fname
             n = 0
             for msg, f, ln in self.parser.doc():
-                msg_list = msg.split("\n")
+                msg_list = statemachine.string2lines(msg, tab_width,
+                                                     convert_whitespace=True)
                 if "debug" in self.options:
                     lines = [
                         "", "",  ".. code-block:: rst",
-- 
2.48.1

[RFC v2 21/38] scripts/get_abi.pl: Add filtering capabilities to rest output

[Mauro Carvalho Chehab]

This way, Sphinx ABI extension can parse symbols only once, while
keep displaying results in separate files.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index b57f46b91828..6d7fa8fbca34 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -173,6 +173,7 @@ class AbiParser:
                         self.data[fdata.key] = {
                             "what": [content],
                             "file": [fdata.file_ref],
+                            "path": fdata.ftype,
                             "line_no": fdata.ln,
                         }
 
@@ -195,8 +196,6 @@ class AbiParser:
                 if new_what:
                     fdata.label = ""
 
-                    self.data[fdata.key]["type"] = fdata.ftype
-
                     if "description" in self.data[fdata.key]:
                         self.data[fdata.key]["description"] += "\n\n"
 
@@ -312,6 +311,7 @@ class AbiParser:
         fdata.nametag = {}
         fdata.nametag["what"] = [f"File {path}/{basename}"]
         fdata.nametag["type"] = "File"
+        fdata.nametag["path"] = fdata.ftype
         fdata.nametag["file"] = [fdata.file_ref]
         fdata.nametag["line_no"] = 1
         fdata.nametag["description"] = ""
@@ -441,7 +441,8 @@ class AbiParser:
 
         return new_desc + "\n\n"
 
-    def doc(self, output_in_txt=False, show_file=True):
+    def doc(self, output_in_txt=False, show_symbols=True, show_file=True,
+            filter_path=None):
         """Print ABI at stdout"""
 
         part = None
@@ -449,12 +450,20 @@ class AbiParser:
                              key=lambda x: (x[1].get("type", ""),
                                             x[1].get("what"))):
 
-            wtype = v.get("type", "Var")
+            wtype = v.get("type", "Symbol")
             file_ref = v.get("file")
             names = v.get("what", [""])
 
-            if not show_file and wtype == "File":
-                continue
+            if wtype == "File":
+                if not show_file:
+                    continue
+            else:
+                if not show_symbols:
+                    continue
+
+            if filter_path:
+                if v.get("path") != filter_path:
+                    continue
 
             msg = ""
 
@@ -616,7 +625,11 @@ class AbiRest:
                                  "It not used, output will contain dynamically"
                                  " generated cross references when possible.")
         parser.add_argument("--no-file", action="store_true",
-                            help="Don't the files section")
+                            help="Don't show files section")
+        parser.add_argument("--no-symbols", action="store_true",
+                            help="Don't show symbols section")
+        parser.add_argument("--filter",
+                            help="Filter a section of ABI (e..g stable, testing, obsolete, removed)")
         parser.add_argument("--show-hints", help="Show-hints")
 
         parser.set_defaults(func=self.run)
@@ -628,7 +641,12 @@ class AbiRest:
         parser.parse_abi()
         parser.check_issues()
 
-        for t in parser.doc(args.raw, not args.no_file):
+        for t in parser.doc(args.raw, show_file=not args.no_file,
+                            show_symbols=not args.no_symbols,
+                            filter_path=args.filter):
+
+            # As line number is returned at the tuple, artifically place
+            # them as a comment tag if one wants to debug troubles there
             if args.enable_lineno:
                 print (f".. LINENO {t[1]}#{t[2]}\n\n")
 
-- 
2.48.1

[RFC v2 22/38] scripts/get_abi.pl: add support to parse ABI README file

[Mauro Carvalho Chehab]

The Documentation/ABI/README file is currently outside the
documentation tree. Yet, it may still provide some useful
information. Add it to the documentation parsing.

As a plus, this avoids a warning when detecting missing
cross-references.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 6d7fa8fbca34..3cde278b8e4c 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -276,6 +276,16 @@ class AbiParser:
         if content:
             self.warn(fdata, "Unexpected content", line)
 
+    def parse_readme(self, nametag, fname):
+        """Parse ABI README file"""
+
+        with open(fname, "r", encoding="utf8", errors="backslashreplace") as fp:
+            nametag["description"] = "```\n"
+            for line in fp:
+                nametag["description"] += "  " + line
+
+            nametag["description"] += "```\n"
+
     def parse_file(self, fname, path, basename):
         """Parse a single file"""
 
@@ -322,6 +332,10 @@ class AbiParser:
         if self.debug & DEBUG_WHAT_OPEN:
             self.log.debug("Opening file %s", fname)
 
+        if basename == "README":
+            self.parse_readme(fdata.nametag, fname)
+            return
+
         with open(fname, "r", encoding="utf8", errors="backslashreplace") as fp:
             for line in fp:
                 fdata.ln += 1
@@ -357,9 +371,6 @@ class AbiParser:
 
                 basename = os.path.basename(name)
 
-                if basename == "README":
-                    continue
-
                 if basename.startswith("."):
                     continue
 
@@ -462,8 +473,12 @@ class AbiParser:
                     continue
 
             if filter_path:
-                if v.get("path") != filter_path:
-                    continue
+                if filter_path == "README":
+                    if not names[0].endswith("README"):
+                        continue
+                else:
+                    if v.get("path") != filter_path:
+                        continue
 
             msg = ""
 
-- 
2.48.1

[RFC v2 23/38] docs: sphinx/kernel_abi: parse ABI files only once

[Mauro Carvalho Chehab]

Right now, the logic parses ABI files on 4 steps, one for each
directory. While this is fine in principle, by doing that, not
all symbol cross-references will be created.

Change the logic to do the parsing only once in order to get
a global dictionary to be used when creating ABI cross-references.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/admin-guide/abi-obsolete.rst |   2 +-
 Documentation/admin-guide/abi-removed.rst  |   2 +-
 Documentation/admin-guide/abi-stable.rst   |   2 +-
 Documentation/admin-guide/abi-testing.rst  |   2 +-
 Documentation/sphinx/kernel_abi.py         | 115 ++++++++++++---------
 scripts/get_abi.py                         |  22 ++--
 6 files changed, 81 insertions(+), 64 deletions(-)

diff --git a/Documentation/admin-guide/abi-obsolete.rst b/Documentation/admin-guide/abi-obsolete.rst
index 6d4d9ab7b8c3..bdef91d2cea4 100644
--- a/Documentation/admin-guide/abi-obsolete.rst
+++ b/Documentation/admin-guide/abi-obsolete.rst
@@ -9,4 +9,4 @@ marked to be removed at some later point in time.
 The description of the interface will document the reason why it is
 obsolete and when it can be expected to be removed.
 
-.. kernel-abi:: ABI/obsolete
+.. kernel-abi:: obsolete
diff --git a/Documentation/admin-guide/abi-removed.rst b/Documentation/admin-guide/abi-removed.rst
index 9fc78af6f077..bea0608b8442 100644
--- a/Documentation/admin-guide/abi-removed.rst
+++ b/Documentation/admin-guide/abi-removed.rst
@@ -3,4 +3,4 @@
 ABI removed symbols
 ===================
 
-.. kernel-abi:: ABI/removed
+.. kernel-abi:: removed
diff --git a/Documentation/admin-guide/abi-stable.rst b/Documentation/admin-guide/abi-stable.rst
index c47c2a295865..33637c0d4fd5 100644
--- a/Documentation/admin-guide/abi-stable.rst
+++ b/Documentation/admin-guide/abi-stable.rst
@@ -12,4 +12,4 @@ for at least 2 years.
 Most interfaces (like syscalls) are expected to never change and always
 be available.
 
-.. kernel-abi:: ABI/stable
+.. kernel-abi:: stable
diff --git a/Documentation/admin-guide/abi-testing.rst b/Documentation/admin-guide/abi-testing.rst
index 40b31985e587..55054985a8ff 100644
--- a/Documentation/admin-guide/abi-testing.rst
+++ b/Documentation/admin-guide/abi-testing.rst
@@ -18,4 +18,4 @@ Programs that use these interfaces are strongly encouraged to add their
 name to the description of these interfaces, so that the kernel
 developers can easily notify them if any changes occur.
 
-.. kernel-abi:: ABI/testing
+.. kernel-abi:: testing
diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 2408ba7a3981..6d1b34e44d46 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -49,6 +49,13 @@ from get_abi import AbiParser
 
 __version__ = "1.0"
 
+logger = logging.getLogger('kernel_abi')
+path = os.path.join(srctree, "Documentation/ABI")
+
+# Parse ABI symbols only once
+kernel_abi = AbiParser(path, logger=logger)
+kernel_abi.parse_abi()
+kernel_abi.check_issues()
 
 def setup(app):
 
@@ -64,14 +71,15 @@ class KernelCmd(Directive):
     u"""KernelABI (``kernel-abi``) directive"""
 
     required_arguments = 1
-    optional_arguments = 2
+    optional_arguments = 3
     has_content = False
     final_argument_whitespace = True
-    logger = logging.getLogger('kernel_abi')
     parser = None
 
     option_spec = {
         "debug": directives.flag,
+        "no-symbols": directives.flag,
+        "no-files":  directives.flag,
     }
 
     def run(self):
@@ -79,62 +87,67 @@ class KernelCmd(Directive):
         if not doc.settings.file_insertion_enabled:
             raise self.warning("docutils: file insertion disabled")
 
-        path = os.path.join(srctree, "Documentation", self.arguments[0])
-        self.parser = AbiParser(path, logger=self.logger)
-        self.parser.parse_abi()
-        self.parser.check_issues()
-
-        node = self.nested_parse(None, self.arguments[0])
-        return node
-
-    def nested_parse(self, data, fname):
         env = self.state.document.settings.env
         content = ViewList()
         node = nodes.section()
 
-        if data is not None:
-            # Handles the .rst file
-            for line in data.split("\n"):
-                content.append(line, fname, 0)
+        abi_type = self.arguments[0]
 
-            self.do_parse(content, node)
+        if "no-symbols" in self.options:
+            show_symbols = False
+        else:
+            show_symbols = True
 
+        if "no-files" in self.options:
+            show_file = False
+        else:
+            show_file = True
+
+        tab_width = self.options.get('tab-width',
+                                     self.state.document.settings.tab_width)
+
+        old_f = None
+        n = 0
+        n_sym = 0
+        for msg, f, ln in kernel_abi.doc(show_file=show_file,
+                                            show_symbols=show_symbols,
+                                            filter_path=abi_type):
+            n_sym += 1
+            msg_list = statemachine.string2lines(msg, tab_width,
+                                                 convert_whitespace=True)
+            if "debug" in self.options:
+                lines = [
+                    "", "",  ".. code-block:: rst",
+                    "    :linenos:", ""
+                ]
+                for m in msg_list:
+                    lines.append("    " + m)
+            else:
+                lines = msg_list
+
+            for line in lines:
+                # sphinx counts lines from 0
+                content.append(line, f, ln - 1)
+                n += 1
+
+            if f != old_f:
+                # Add the file to Sphinx build dependencies
+                env.note_dependency(os.path.abspath(f))
+
+                old_f = f
+
+            # Sphinx doesn't like to parse big messages. So, let's
+            # add content symbol by symbol
+            if content:
+                self.do_parse(content, node)
+                content = ViewList()
+
+        if show_symbols and not show_file:
+            logger.verbose("%s ABI: %i symbols (%i ReST lines)" % (abi_type, n_sym, n))
+        elif not show_symbols and show_file:
+            logger.verbose("%s ABI: %i files (%i ReST lines)" % (abi_type, n_sym, n))
         else:
-            # Handles the ABI parser content, symbol by symbol
-
-            old_f = fname
-            n = 0
-            for msg, f, ln in self.parser.doc():
-                msg_list = statemachine.string2lines(msg, tab_width,
-                                                     convert_whitespace=True)
-                if "debug" in self.options:
-                    lines = [
-                        "", "",  ".. code-block:: rst",
-                        "    :linenos:", ""
-                    ]
-                    for m in msg_list:
-                        lines.append("    " + m)
-                else:
-                    lines = msg_list
-
-                for line in lines:
-                    # sphinx counts lines from 0
-                    content.append(line, f, ln - 1)
-                    n += 1
-
-                if f != old_f:
-                    # Add the file to Sphinx build dependencies
-                    env.note_dependency(os.path.abspath(f))
-
-                    old_f = f
-
-                # Sphinx doesn't like to parse big messages. So, let's
-                # add content symbol by symbol
-                if content:
-                    self.do_parse(content, node)
-                    content = ViewList()
-
-            self.logger.info("%s: parsed %i lines" % (fname, n))
+            logger.verbose("%s ABI: %i data (%i ReST lines)" % (abi_type, n_sym, n))
 
         return node.children
 
diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 3cde278b8e4c..7435ed1071b8 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -279,12 +279,20 @@ class AbiParser:
     def parse_readme(self, nametag, fname):
         """Parse ABI README file"""
 
+        nametag["what"] = ["ABI file contents"]
+        nametag["path"] = "README"
         with open(fname, "r", encoding="utf8", errors="backslashreplace") as fp:
-            nametag["description"] = "```\n"
             for line in fp:
-                nametag["description"] += "  " + line
+                match = self.re_tag.match(line)
+                if match:
+                    new = match.group(1).lower()
 
-            nametag["description"] += "```\n"
+                    match = self.re_valid.search(new)
+                    if match:
+                        nametag["description"] += "\n:" + line
+                        continue
+
+                nametag["description"] += line
 
     def parse_file(self, fname, path, basename):
         """Parse a single file"""
@@ -473,12 +481,8 @@ class AbiParser:
                     continue
 
             if filter_path:
-                if filter_path == "README":
-                    if not names[0].endswith("README"):
-                        continue
-                else:
-                    if v.get("path") != filter_path:
-                        continue
+                if v.get("path") != filter_path:
+                    continue
 
             msg = ""
 
-- 
2.48.1

[RFC v2 24/38] docs: admin-guide/abi: split files from symbols

[Mauro Carvalho Chehab]

Now that get_abi has gained support for filtering its output,
split ABI symbols from files at the html output.

That makes pages smaller and easier to navigate.

As an additional bonus, as it will paralelize files handling,
it gives an additional performance improvement.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/admin-guide/abi-obsolete-files.rst |  7 +++++++
 Documentation/admin-guide/abi-obsolete.rst       |  1 +
 Documentation/admin-guide/abi-readme-file.rst    |  6 ++++++
 Documentation/admin-guide/abi-removed-files.rst  |  7 +++++++
 Documentation/admin-guide/abi-removed.rst        |  1 +
 Documentation/admin-guide/abi-stable-files.rst   |  7 +++++++
 Documentation/admin-guide/abi-stable.rst         |  1 +
 Documentation/admin-guide/abi-testing-files.rst  |  7 +++++++
 Documentation/admin-guide/abi-testing.rst        |  1 +
 Documentation/admin-guide/abi.rst                | 15 +++++++++++++++
 10 files changed, 53 insertions(+)
 create mode 100644 Documentation/admin-guide/abi-obsolete-files.rst
 create mode 100644 Documentation/admin-guide/abi-readme-file.rst
 create mode 100644 Documentation/admin-guide/abi-removed-files.rst
 create mode 100644 Documentation/admin-guide/abi-stable-files.rst
 create mode 100644 Documentation/admin-guide/abi-testing-files.rst

diff --git a/Documentation/admin-guide/abi-obsolete-files.rst b/Documentation/admin-guide/abi-obsolete-files.rst
new file mode 100644
index 000000000000..3061a916b4b5
--- /dev/null
+++ b/Documentation/admin-guide/abi-obsolete-files.rst
@@ -0,0 +1,7 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Obsolete ABI Files
+==================
+
+.. kernel-abi:: obsolete
+   :no-symbols:
diff --git a/Documentation/admin-guide/abi-obsolete.rst b/Documentation/admin-guide/abi-obsolete.rst
index bdef91d2cea4..640f3903e847 100644
--- a/Documentation/admin-guide/abi-obsolete.rst
+++ b/Documentation/admin-guide/abi-obsolete.rst
@@ -10,3 +10,4 @@ The description of the interface will document the reason why it is
 obsolete and when it can be expected to be removed.
 
 .. kernel-abi:: obsolete
+   :no-files:
diff --git a/Documentation/admin-guide/abi-readme-file.rst b/Documentation/admin-guide/abi-readme-file.rst
new file mode 100644
index 000000000000..6172e4ccbda2
--- /dev/null
+++ b/Documentation/admin-guide/abi-readme-file.rst
@@ -0,0 +1,6 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+ABI README
+==========
+
+.. kernel-abi:: README
diff --git a/Documentation/admin-guide/abi-removed-files.rst b/Documentation/admin-guide/abi-removed-files.rst
new file mode 100644
index 000000000000..f1bdfadd2ec4
--- /dev/null
+++ b/Documentation/admin-guide/abi-removed-files.rst
@@ -0,0 +1,7 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Removed ABI Files
+=================
+
+.. kernel-abi:: removed
+   :no-symbols:
diff --git a/Documentation/admin-guide/abi-removed.rst b/Documentation/admin-guide/abi-removed.rst
index bea0608b8442..88832d3eacd6 100644
--- a/Documentation/admin-guide/abi-removed.rst
+++ b/Documentation/admin-guide/abi-removed.rst
@@ -4,3 +4,4 @@ ABI removed symbols
 ===================
 
 .. kernel-abi:: removed
+   :no-files:
diff --git a/Documentation/admin-guide/abi-stable-files.rst b/Documentation/admin-guide/abi-stable-files.rst
new file mode 100644
index 000000000000..f867738fc178
--- /dev/null
+++ b/Documentation/admin-guide/abi-stable-files.rst
@@ -0,0 +1,7 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Stable ABI Files
+================
+
+.. kernel-abi:: stable
+   :no-symbols:
diff --git a/Documentation/admin-guide/abi-stable.rst b/Documentation/admin-guide/abi-stable.rst
index 33637c0d4fd5..528c68401f4b 100644
--- a/Documentation/admin-guide/abi-stable.rst
+++ b/Documentation/admin-guide/abi-stable.rst
@@ -13,3 +13,4 @@ Most interfaces (like syscalls) are expected to never change and always
 be available.
 
 .. kernel-abi:: stable
+   :no-files:
diff --git a/Documentation/admin-guide/abi-testing-files.rst b/Documentation/admin-guide/abi-testing-files.rst
new file mode 100644
index 000000000000..1da868e42fdb
--- /dev/null
+++ b/Documentation/admin-guide/abi-testing-files.rst
@@ -0,0 +1,7 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Testing ABI Files
+=================
+
+.. kernel-abi:: testing
+   :no-symbols:
diff --git a/Documentation/admin-guide/abi-testing.rst b/Documentation/admin-guide/abi-testing.rst
index 55054985a8ff..6153ebd38e2d 100644
--- a/Documentation/admin-guide/abi-testing.rst
+++ b/Documentation/admin-guide/abi-testing.rst
@@ -19,3 +19,4 @@ name to the description of these interfaces, so that the kernel
 developers can easily notify them if any changes occur.
 
 .. kernel-abi:: testing
+   :no-files:
diff --git a/Documentation/admin-guide/abi.rst b/Documentation/admin-guide/abi.rst
index 64e772bde943..15a2dcb1388c 100644
--- a/Documentation/admin-guide/abi.rst
+++ b/Documentation/admin-guide/abi.rst
@@ -4,6 +4,9 @@
 Linux ABI description
 =====================
 
+ABI symbols
+-----------
+
 .. toctree::
    :maxdepth: 2
 
@@ -11,3 +14,15 @@ Linux ABI description
    abi-testing
    abi-obsolete
    abi-removed
+
+ABI files
+---------
+
+.. toctree::
+   :maxdepth: 2
+
+   abi-readme-file
+   abi-stable-files
+   abi-testing-files
+   abi-obsolete-files
+   abi-removed-files
-- 
2.48.1

[RFC v2 25/38] docs: sphinx/automarkup: add cross-references for ABI

[Mauro Carvalho Chehab]

Now that all ABI files are handled together, we can add a feature
at automarkup for it to generate cross-references for ABI symbols.

The cross-reference logic can produce references for all existing
files, except for README (as this is not parsed).

For symbols, they need to be an exact match of what it is
described at the docs, which is not always true due to wildcards.

If symbols at /sys /proc and /config are identical, a cross-reference
will be used.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/automarkup.py | 45 ++++++++++++++++++++++++++++++
 scripts/get_abi.py                 | 11 ++++++++
 2 files changed, 56 insertions(+)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index a413f8dd5115..7d91c39b4ca6 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -11,6 +11,8 @@ from sphinx.errors import NoUri
 import re
 from itertools import chain
 
+from kernel_abi import kernel_abi
+
 #
 # Python 2 lacks re.ASCII...
 #
@@ -48,6 +50,8 @@ RE_typedef = re.compile(r'\b(typedef)\s+([a-zA-Z_]\w+)', flags=ascii_p3)
 # an optional extension
 #
 RE_doc = re.compile(r'(\bDocumentation/)?((\.\./)*[\w\-/]+)\.(rst|txt)')
+RE_abi_file = re.compile(r'(\bDocumentation/ABI/[\w\-/]+)')
+RE_abi_symbol = re.compile(r'(\b/(sys|config|proc)/[\w\-/]+)')
 
 RE_namespace = re.compile(r'^\s*..\s*c:namespace::\s*(\S+)\s*$')
 
@@ -84,10 +88,14 @@ def markup_refs(docname, app, node):
     # Associate each regex with the function that will markup its matches
     #
     markup_func_sphinx2 = {RE_doc: markup_doc_ref,
+                           RE_abi_file: markup_abi_ref,
+                           RE_abi_symbol: markup_abi_ref,
                            RE_function: markup_c_ref,
                            RE_generic_type: markup_c_ref}
 
     markup_func_sphinx3 = {RE_doc: markup_doc_ref,
+                           RE_abi_file: markup_abi_ref,
+                           RE_abi_symbol: markup_abi_ref,
                            RE_function: markup_func_ref_sphinx3,
                            RE_struct: markup_c_ref,
                            RE_union: markup_c_ref,
@@ -270,6 +278,43 @@ def markup_doc_ref(docname, app, match):
     else:
         return nodes.Text(match.group(0))
 
+#
+# Try to replace a documentation reference of the form Documentation/ABI/...
+# with a cross reference to that page
+#
+def markup_abi_ref(docname, app, match):
+    stddom = app.env.domains['std']
+    #
+    # Go through the dance of getting an xref out of the std domain
+    #
+    fname = match.group(1)
+    target = kernel_abi.xref(fname)
+
+    # Kernel ABI doesn't describe such file or symbol
+    if not target:
+        return nodes.Text(match.group(0))
+
+    pxref = addnodes.pending_xref('', refdomain = 'std', reftype = 'ref',
+                                  reftarget = target, modname = None,
+                                  classname = None, refexplicit = False)
+
+    #
+    # XXX The Latex builder will throw NoUri exceptions here,
+    # work around that by ignoring them.
+    #
+    try:
+        xref = stddom.resolve_xref(app.env, docname, app.builder, 'ref',
+                                   target, pxref, None)
+    except NoUri:
+        xref = None
+    #
+    # Return the xref if we got it; otherwise just return the plain text.
+    #
+    if xref:
+        return xref
+    else:
+        return nodes.Text(match.group(0))
+
 def get_c_namespace(app, docname):
     source = app.env.doc2path(docname)
     with open(source) as f:
diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 7435ed1071b8..faae51201504 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -404,6 +404,17 @@ class AbiParser:
 
         return desc + "\n\n"
 
+    def xref(self, fname):
+        """
+        Converts a Documentation/ABI + basename into a ReST cross-reference
+        """
+
+        xref = self.file_refs.get(fname)
+        if not xref:
+            return None
+        else:
+            return xref
+
     def desc_rst(self, desc):
         """Enrich ReST output by creating cross-references"""
 
-- 
2.48.1

[RFC v2 26/38] docs: sphinx/kernel_abi: avoid warnings during Sphinx module init

[Mauro Carvalho Chehab]

Sphinx logging system doesn't like warnings during module load,
as it understands that such logs are produced at the wrong time:

	WARNING: while setting up extension automarkup: /sys/devices/system/cpu/cpuX/topology/physical_package_id is defined 2 times: /new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:27; /new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70
	WARNING: while setting up extension automarkup: /sys/devices/system/cpu/cpuX/topology/ppin is defined 2 times: /new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:89; /new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70

So, use a function to allocate/process ABI files and use it to
be called at kernel_abi.py, as automarkup also needs it to
produce the right cross-references.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/automarkup.py |  4 +++-
 Documentation/sphinx/kernel_abi.py | 23 +++++++++++++++++++----
 2 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index 7d91c39b4ca6..22defc18d6d2 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -11,7 +11,7 @@ from sphinx.errors import NoUri
 import re
 from itertools import chain
 
-from kernel_abi import kernel_abi
+from kernel_abi import get_kernel_abi
 
 #
 # Python 2 lacks re.ASCII...
@@ -287,6 +287,8 @@ def markup_abi_ref(docname, app, match):
     #
     # Go through the dance of getting an xref out of the std domain
     #
+    kernel_abi = get_kernel_abi()
+
     fname = match.group(1)
     target = kernel_abi.xref(fname)
 
diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
index 6d1b34e44d46..379a877b75f9 100644
--- a/Documentation/sphinx/kernel_abi.py
+++ b/Documentation/sphinx/kernel_abi.py
@@ -52,10 +52,23 @@ __version__ = "1.0"
 logger = logging.getLogger('kernel_abi')
 path = os.path.join(srctree, "Documentation/ABI")
 
-# Parse ABI symbols only once
-kernel_abi = AbiParser(path, logger=logger)
-kernel_abi.parse_abi()
-kernel_abi.check_issues()
+_kernel_abi = None
+
+def get_kernel_abi():
+    u"""
+    Initialize kernel_abi global var, if not initialized yet.
+
+    This is needed to avoid warnings during Sphinx module initialization.
+    """
+    global _kernel_abi
+
+    if not _kernel_abi:
+        # Parse ABI symbols only once
+        _kernel_abi = AbiParser(path, logger=logger)
+        _kernel_abi.parse_abi()
+        _kernel_abi.check_issues()
+
+    return _kernel_abi
 
 def setup(app):
 
@@ -83,6 +96,8 @@ class KernelCmd(Directive):
     }
 
     def run(self):
+        kernel_abi = get_kernel_abi()
+
         doc = self.state.document
         if not doc.settings.file_insertion_enabled:
             raise self.warning("docutils: file insertion disabled")
-- 
2.48.1

[RFC v2 27/38] scripts/get_abi.py: Rename title name for ABI files

[Mauro Carvalho Chehab]

This makes them look better when generating cross-references.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index faae51201504..1c9c6b23a6fb 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -327,7 +327,7 @@ class AbiParser:
         fdata.ftype = path.split("/")[0]
 
         fdata.nametag = {}
-        fdata.nametag["what"] = [f"File {path}/{basename}"]
+        fdata.nametag["what"] = [f"ABI file {path}/{basename}"]
         fdata.nametag["type"] = "File"
         fdata.nametag["path"] = fdata.ftype
         fdata.nametag["file"] = [fdata.file_ref]
-- 
2.48.1

[RFC v2 28/38] docs: media: Allow creating cross-references for RC ABI

[Mauro Carvalho Chehab]

Now that Documentation/ABI is processed by automarkup, let it
generate cross-references for RC sysfs devnodes.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/sphinx/automarkup.py              | 17 +++++++++++++----
 .../userspace-api/media/rc/rc-sysfs-nodes.rst   |  2 +-
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/Documentation/sphinx/automarkup.py b/Documentation/sphinx/automarkup.py
index 22defc18d6d2..03bb7020f5cb 100644
--- a/Documentation/sphinx/automarkup.py
+++ b/Documentation/sphinx/automarkup.py
@@ -88,13 +88,13 @@ def markup_refs(docname, app, node):
     # Associate each regex with the function that will markup its matches
     #
     markup_func_sphinx2 = {RE_doc: markup_doc_ref,
-                           RE_abi_file: markup_abi_ref,
+                           RE_abi_file: markup_abi_file_ref,
                            RE_abi_symbol: markup_abi_ref,
                            RE_function: markup_c_ref,
                            RE_generic_type: markup_c_ref}
 
     markup_func_sphinx3 = {RE_doc: markup_doc_ref,
-                           RE_abi_file: markup_abi_ref,
+                           RE_abi_file: markup_abi_file_ref,
                            RE_abi_symbol: markup_abi_ref,
                            RE_function: markup_func_ref_sphinx3,
                            RE_struct: markup_c_ref,
@@ -279,10 +279,10 @@ def markup_doc_ref(docname, app, match):
         return nodes.Text(match.group(0))
 
 #
-# Try to replace a documentation reference of the form Documentation/ABI/...
+# Try to replace a documentation reference for ABI symbols and files
 # with a cross reference to that page
 #
-def markup_abi_ref(docname, app, match):
+def markup_abi_ref(docname, app, match, warning=False):
     stddom = app.env.domains['std']
     #
     # Go through the dance of getting an xref out of the std domain
@@ -294,6 +294,8 @@ def markup_abi_ref(docname, app, match):
 
     # Kernel ABI doesn't describe such file or symbol
     if not target:
+        if warning:
+            kernel_abi.log.warning("%s not found", fname)
         return nodes.Text(match.group(0))
 
     pxref = addnodes.pending_xref('', refdomain = 'std', reftype = 'ref',
@@ -317,6 +319,13 @@ def markup_abi_ref(docname, app, match):
     else:
         return nodes.Text(match.group(0))
 
+#
+# Variant of markup_abi_ref() that warns whan a reference is not found
+#
+def markup_abi_file_ref(docname, app, match):
+    return markup_abi_ref(docname, app, match, warning=True)
+
+
 def get_c_namespace(app, docname):
     source = app.env.doc2path(docname)
     with open(source) as f:
diff --git a/Documentation/userspace-api/media/rc/rc-sysfs-nodes.rst b/Documentation/userspace-api/media/rc/rc-sysfs-nodes.rst
index 34d6a0a1f4d3..70b5966aaff8 100644
--- a/Documentation/userspace-api/media/rc/rc-sysfs-nodes.rst
+++ b/Documentation/userspace-api/media/rc/rc-sysfs-nodes.rst
@@ -6,7 +6,7 @@
 Remote Controller's sysfs nodes
 *******************************
 
-As defined at ``Documentation/ABI/testing/sysfs-class-rc``, those are
+As defined at Documentation/ABI/testing/sysfs-class-rc, those are
 the sysfs nodes that control the Remote Controllers:
 
 
-- 
2.48.1

[RFC v2 29/38] docs: thunderbolt: Allow creating cross-references for ABI

[Mauro Carvalho Chehab]

Now that Documentation/ABI is processed by automarkup, let it
generate cross-references for the corresponding ABI file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/admin-guide/thunderbolt.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/thunderbolt.rst b/Documentation/admin-guide/thunderbolt.rst
index 2ed79f41a411..d0502691dfa1 100644
--- a/Documentation/admin-guide/thunderbolt.rst
+++ b/Documentation/admin-guide/thunderbolt.rst
@@ -28,7 +28,7 @@ should be a userspace tool that handles all the low-level details, keeps
 a database of the authorized devices and prompts users for new connections.
 
 More details about the sysfs interface for Thunderbolt devices can be
-found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``.
+found in Documentation/ABI/testing/sysfs-bus-thunderbolt.
 
 Those users who just want to connect any device without any sort of
 manual work can add following line to
-- 
2.48.1

[RFC v2 30/38] docs: arm: asymmetric-32bit: Allow creating cross-references for ABI

[Mauro Carvalho Chehab]

Now that Documentation/ABI is processed by automarkup, let it
generate cross-references for the corresponding ABI file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/arch/arm64/asymmetric-32bit.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/arch/arm64/asymmetric-32bit.rst b/Documentation/arch/arm64/asymmetric-32bit.rst
index 64a0b505da7d..f15061b1e020 100644
--- a/Documentation/arch/arm64/asymmetric-32bit.rst
+++ b/Documentation/arch/arm64/asymmetric-32bit.rst
@@ -55,7 +55,7 @@ sysfs
 
 The subset of CPUs capable of running 32-bit tasks is described in
 ``/sys/devices/system/cpu/aarch32_el0`` and is documented further in
-``Documentation/ABI/testing/sysfs-devices-system-cpu``.
+Documentation/ABI/testing/sysfs-devices-system-cpu.
 
 **Note:** CPUs are advertised by this file as they are detected and so
 late-onlining of 32-bit-capable CPUs can result in the file contents
-- 
2.48.1

[RFC v2 31/38] docs: arm: generic-counter: Allow creating cross-references for ABI

[Mauro Carvalho Chehab]

Now that Documentation/ABI is processed by automarkup, let it
generate cross-references for the corresponding ABI file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/driver-api/generic-counter.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/driver-api/generic-counter.rst b/Documentation/driver-api/generic-counter.rst
index 71ccc30e586b..e826f16ea43d 100644
--- a/Documentation/driver-api/generic-counter.rst
+++ b/Documentation/driver-api/generic-counter.rst
@@ -467,7 +467,7 @@ Counter sysfs
 Translates counter data to the standard Counter sysfs interface format
 and vice versa.
 
-Please refer to the ``Documentation/ABI/testing/sysfs-bus-counter`` file
+Please refer to the Documentation/ABI/testing/sysfs-bus-counter file
 for a detailed breakdown of the available Generic Counter interface
 sysfs attributes.
 
@@ -483,7 +483,7 @@ Sysfs Interface
 Several sysfs attributes are generated by the Generic Counter interface,
 and reside under the ``/sys/bus/counter/devices/counterX`` directory,
 where ``X`` is to the respective counter device id. Please see
-``Documentation/ABI/testing/sysfs-bus-counter`` for detailed information
+Documentation/ABI/testing/sysfs-bus-counter for detailed information
 on each Generic Counter interface sysfs attribute.
 
 Through these sysfs attributes, programs and scripts may interact with
-- 
2.48.1

[RFC v2 32/38] docs: iio: Allow creating cross-references ABI

[Mauro Carvalho Chehab]

Now that Documentation/ABI is processed by automarkup, let it
generate cross-references for the corresponding ABI file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/driver-api/iio/core.rst | 2 +-
 Documentation/iio/iio_devbuf.rst      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/driver-api/iio/core.rst b/Documentation/driver-api/iio/core.rst
index dfe438dc91a7..42b580fb2989 100644
--- a/Documentation/driver-api/iio/core.rst
+++ b/Documentation/driver-api/iio/core.rst
@@ -60,7 +60,7 @@ directory.  Common attributes are:
 * :file:`sampling_frequency_available`, available discrete set of sampling
   frequency values for device.
 * Available standard attributes for IIO devices are described in the
-  :file:`Documentation/ABI/testing/sysfs-bus-iio` file in the Linux kernel
+  :file:Documentation/ABI/testing/sysfs-bus-iio file in the Linux kernel
   sources.
 
 IIO device channels
diff --git a/Documentation/iio/iio_devbuf.rst b/Documentation/iio/iio_devbuf.rst
index 9919e4792d0e..dca1f0200b0d 100644
--- a/Documentation/iio/iio_devbuf.rst
+++ b/Documentation/iio/iio_devbuf.rst
@@ -148,5 +148,5 @@ applied), however there are corner cases in which the buffered data may be found
 in a processed form. Please note that these corner cases are not addressed by
 this documentation.
 
-Please see ``Documentation/ABI/testing/sysfs-bus-iio`` for a complete
+Please see Documentation/ABI/testing/sysfs-bus-iio for a complete
 description of the attributes.
-- 
2.48.1

[RFC v2 33/38] docs: networking: Allow creating cross-references statistics ABI

[Mauro Carvalho Chehab]

Now that Documentation/ABI is processed by automarkup, let it
generate cross-references for the corresponding ABI file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/networking/statistics.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/networking/statistics.rst b/Documentation/networking/statistics.rst
index 75e017dfa825..518284e287b0 100644
--- a/Documentation/networking/statistics.rst
+++ b/Documentation/networking/statistics.rst
@@ -143,7 +143,7 @@ reading multiple stats as it internally performs a full dump of
 and reports only the stat corresponding to the accessed file.
 
 Sysfs files are documented in
-`Documentation/ABI/testing/sysfs-class-net-statistics`.
+Documentation/ABI/testing/sysfs-class-net-statistics.
 
 
 netlink
-- 
2.48.1

[RFC v2 34/38] docs: submit-checklist: Allow creating cross-references for ABI README

[Mauro Carvalho Chehab]

Now that Documentation/ABI is processed by automarkup, let it
generate cross-references for the ABI README file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/process/submit-checklist.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/process/submit-checklist.rst b/Documentation/process/submit-checklist.rst
index e531dd504b6c..827566872567 100644
--- a/Documentation/process/submit-checklist.rst
+++ b/Documentation/process/submit-checklist.rst
@@ -52,7 +52,7 @@ Provide documentation
 4) All new module parameters are documented with ``MODULE_PARM_DESC()``
 
 5) All new userspace interfaces are documented in ``Documentation/ABI/``.
-   See ``Documentation/ABI/README`` for more information.
+   See Documentation/ABI/README for more information.
    Patches that change userspace interfaces should be CCed to
    linux-api@vger.kernel.org.
 
-- 
2.48.1

[RFC v2 35/38] docs: translations: Allow creating cross-references for ABI README

[Mauro Carvalho Chehab]

Update translations to also generate cross references for the ABI
readme file.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/translations/it_IT/process/submit-checklist.rst | 2 +-
 Documentation/translations/sp_SP/process/submit-checklist.rst | 2 +-
 Documentation/translations/zh_CN/process/submit-checklist.rst | 2 +-
 Documentation/translations/zh_TW/process/submit-checklist.rst | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Documentation/translations/it_IT/process/submit-checklist.rst b/Documentation/translations/it_IT/process/submit-checklist.rst
index 692be4af9c9b..3308c3084c60 100644
--- a/Documentation/translations/it_IT/process/submit-checklist.rst
+++ b/Documentation/translations/it_IT/process/submit-checklist.rst
@@ -58,7 +58,7 @@ Fornite documentazione
 4) Tutti i nuovi parametri dei moduli sono documentati con ``MODULE_PARM_DESC()``.
 
 5) Tutte le nuove interfacce verso lo spazio utente sono documentate in
-    ``Documentation/ABI/``.  Leggete ``Documentation/ABI/README`` per maggiori
+    ``Documentation/ABI/``.  Leggete Documentation/ABI/README per maggiori
     informazioni.  Le patch che modificano le interfacce utente dovrebbero
     essere inviate in copia anche a linux-api@vger.kernel.org.
 
diff --git a/Documentation/translations/sp_SP/process/submit-checklist.rst b/Documentation/translations/sp_SP/process/submit-checklist.rst
index 0d6651f9d871..3087ba80e818 100644
--- a/Documentation/translations/sp_SP/process/submit-checklist.rst
+++ b/Documentation/translations/sp_SP/process/submit-checklist.rst
@@ -97,7 +97,7 @@ y en otros lugares con respecto al envío de parches del kernel de Linux.
     ``MODULE_PARM_DESC()``.
 
 18) Todas las nuevas interfaces de espacio de usuario están documentadas
-    en ``Documentation/ABI/``. Consulte ``Documentation/ABI/README`` para
+    en ``Documentation/ABI/``. Consulte Documentation/ABI/README para
     obtener más información. Los parches que cambian las interfaces del
     espacio de usuario deben ser CCed a linux-api@vger.kernel.org.
 
diff --git a/Documentation/translations/zh_CN/process/submit-checklist.rst b/Documentation/translations/zh_CN/process/submit-checklist.rst
index 10536b74aeec..5c20a7f0936b 100644
--- a/Documentation/translations/zh_CN/process/submit-checklist.rst
+++ b/Documentation/translations/zh_CN/process/submit-checklist.rst
@@ -82,7 +82,7 @@ Linux内核补丁提交检查单
 17) 所有新的模块参数都记录在 ``MODULE_PARM_DESC()``
 
 18) 所有新的用户空间接口都记录在 ``Documentation/ABI/`` 中。有关详细信息，
-    请参阅 ``Documentation/ABI/README`` 。更改用户空间接口的补丁应该抄送
+    请参阅 Documentation/ABI/README 。更改用户空间接口的补丁应该抄送
     linux-api@vger.kernel.org。
 
 19) 已通过至少注入slab和page分配失败进行检查。请参阅 ``Documentation/fault-injection/`` 。
diff --git a/Documentation/translations/zh_TW/process/submit-checklist.rst b/Documentation/translations/zh_TW/process/submit-checklist.rst
index 0ecb187753e4..b09d29b0cc1b 100644
--- a/Documentation/translations/zh_TW/process/submit-checklist.rst
+++ b/Documentation/translations/zh_TW/process/submit-checklist.rst
@@ -85,7 +85,7 @@ Linux內核補丁提交檢查單
 17) 所有新的模塊參數都記錄在 ``MODULE_PARM_DESC()``
 
 18) 所有新的用戶空間接口都記錄在 ``Documentation/ABI/`` 中。有關詳細信息，
-    請參閱 ``Documentation/ABI/README`` 。更改用戶空間接口的補丁應該抄送
+    請參閱 Documentation/ABI/README 。更改用戶空間接口的補丁應該抄送
     linux-api@vger.kernel.org。
 
 19) 已通過至少注入slab和page分配失敗進行檢查。請參閱 ``Documentation/fault-injection/`` 。
-- 
2.48.1

[RFC v2 36/38] docs: ABI: drop two duplicate symbols

[Mauro Carvalho Chehab]

As warned by get_abi.py, there are two symbols that are
defined twice:

WARNING: /sys/devices/system/cpu/cpuX/topology/physical_package_id is defined 2 times: \
	/new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:27; \
	/new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70
WARNING: /sys/devices/system/cpu/cpuX/topology/ppin is defined 2 times: \
	/new_devel/v4l/docs/Documentation/ABI/stable/sysfs-devices-system-cpu:89; \
	/new_devel/v4l/docs/Documentation/ABI/testing/sysfs-devices-system-cpu:70

As the documentation at testing/sysfs-devices-system-cpu, drop
the duplicated one from stable.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/ABI/stable/sysfs-devices-system-cpu | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/Documentation/ABI/stable/sysfs-devices-system-cpu b/Documentation/ABI/stable/sysfs-devices-system-cpu
index 902392d7eddf..cf78bd99f6c8 100644
--- a/Documentation/ABI/stable/sysfs-devices-system-cpu
+++ b/Documentation/ABI/stable/sysfs-devices-system-cpu
@@ -24,12 +24,6 @@ Description:	Default value for the Data Stream Control Register (DSCR) on
 		If set by a process it will be inherited by child processes.
 Values:		64 bit unsigned integer (bit field)
 
-What:           /sys/devices/system/cpu/cpuX/topology/physical_package_id
-Description:    physical package id of cpuX. Typically corresponds to a physical
-                socket number, but the actual value is architecture and platform
-                dependent.
-Values:         integer
-
 What:           /sys/devices/system/cpu/cpuX/topology/die_id
 Description:    the CPU die ID of cpuX. Typically it is the hardware platform's
                 identifier (rather than the kernel's). The actual value is
@@ -86,10 +80,6 @@ What:           /sys/devices/system/cpu/cpuX/topology/die_cpus
 Description:    internal kernel map of CPUs within the same die.
 Values:         hexadecimal bitmask.
 
-What:           /sys/devices/system/cpu/cpuX/topology/ppin
-Description:    per-socket protected processor inventory number
-Values:         hexadecimal.
-
 What:           /sys/devices/system/cpu/cpuX/topology/die_cpus_list
 Description:    human-readable list of CPUs within the same die.
                 The format is like 0-3, 8-11, 14,17.
-- 
2.48.1

[RFC v2 37/38] scripts/get_abi.py: add support for undefined ABIs

[Mauro Carvalho Chehab]

The undefined logic is complex and has lots of magic on it.

Implement it, using the same algorithm we have at get_abi.pl. Yet,
some tweaks to optimize performance and to make the code simpler
were added here:
- at the perl version, the tree graph had loops, so we had to
  use BFS to traverse it. On this version, the graph is a tree,
  so, it simplifies the what group for sysfs aliases;
- the logic which splits regular expressions into subgroups
  was re-written to make it faster;
- it may optionally use multiple processes to search for symbol
  matches;
- it has some additional debug levels.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.py | 670 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 669 insertions(+), 1 deletion(-)

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 1c9c6b23a6fb..543bed397c8c 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -13,8 +13,10 @@ import os
 import re
 import sys
 
+from concurrent import futures
 from pprint import pformat
-from random import randrange, seed
+from random import randrange, seed, shuffle
+from datetime import datetime
 
 ABI_DIR = "Documentation/ABI/"
 
@@ -22,6 +24,12 @@ ABI_DIR = "Documentation/ABI/"
 DEBUG_WHAT_PARSING = 1
 DEBUG_WHAT_OPEN = 2
 DEBUG_DUMP_ABI_STRUCTS = 4
+DEBUG_UNDEFINED = 8
+DEBUG_REGEX = 16
+DEBUG_SUBGROUP_MAP = 32
+DEBUG_SUBGROUP_DICT = 64
+DEBUG_SUBGROUP_SIZE = 128
+DEBUG_GRAPH = 256
 
 DEBUG_HELP = """
 Print debug information according with the level(s),
@@ -30,6 +38,13 @@ which is given by the following bitmask:
 1  - enable debug parsing logic
 2  - enable debug messages on file open
 4  - enable debug for ABI parse data
+8  - enable extra debug information to identify troubles
+     with ABI symbols found at the local machine that
+     weren't found on ABI documentation (used only for
+     undefined subcommand)
+16 - enable debug for what to regex conversion
+32 - enable debug for symbol regex subgroups
+64 - enable debug for sysfs graph tree variable
 """
 
 
@@ -638,6 +653,593 @@ class AbiParser:
             print(f"Regular expression /{expr}/ not found.")
 
 
+class AbiRegex(AbiParser):
+    """Extends AbiParser to search ABI nodes with regular expressions"""
+
+    # Escape only ASCII visible characters
+    escape_symbols = r"([\x21-\x29\x2b-\x2d\x3a-\x40\x5c\x60\x7b-\x7e])"
+    leave_others = "others"
+
+    # Tuples with regular expressions to be compiled and replacement data
+    re_whats = [
+        # Drop escape characters that might exist
+        (re.compile("\\\\"), ""),
+
+        # Temporarily escape dot characters
+        (re.compile(r"\."),  "\xf6"),
+
+        # Temporarily change [0-9]+ type of patterns
+        (re.compile(r"\[0\-9\]\+"),  "\xff"),
+
+        # Temporarily change [\d+-\d+] type of patterns
+        (re.compile(r"\[0\-\d+\]"),  "\xff"),
+        (re.compile(r"\[0:\d+\]"),  "\xff"),
+        (re.compile(r"\[(\d+)\]"),  "\xf4\\\\d+\xf5"),
+
+        # Temporarily change [0-9] type of patterns
+        (re.compile(r"\[(\d)\-(\d)\]"),  "\xf4\1-\2\xf5"),
+
+        # Handle multiple option patterns
+        (re.compile(r"[\{\<\[]([\w_]+)(?:[,|]+([\w_]+)){1,}[\}\>\]]"), r"(\1|\2)"),
+
+        # Handle wildcards
+        (re.compile(r"([^\/])\*"), "\\1\\\\w\xf7"),
+        (re.compile(r"/\*/"), "/.*/"),
+        (re.compile(r"/\xf6\xf6\xf6"), "/.*"),
+        (re.compile(r"\<[^\>]+\>"), "\\\\w\xf7"),
+        (re.compile(r"\{[^\}]+\}"), "\\\\w\xf7"),
+        (re.compile(r"\[[^\]]+\]"), "\\\\w\xf7"),
+
+        (re.compile(r"XX+"), "\\\\w\xf7"),
+        (re.compile(r"([^A-Z])[XYZ]([^A-Z])"), "\\1\\\\w\xf7\\2"),
+        (re.compile(r"([^A-Z])[XYZ]$"), "\\1\\\\w\xf7"),
+        (re.compile(r"_[AB]_"), "_\\\\w\xf7_"),
+
+        # Recover [0-9] type of patterns
+        (re.compile(r"\xf4"), "["),
+        (re.compile(r"\xf5"),  "]"),
+
+        # Remove duplicated spaces
+        (re.compile(r"\s+"), r" "),
+
+        # Special case: drop comparison as in:
+        # What: foo = <something>
+        # (this happens on a few IIO definitions)
+        (re.compile(r"\s*\=.*$"), ""),
+
+        # Escape all other symbols
+        (re.compile(escape_symbols), r"\\\1"),
+        (re.compile(r"\\\\"), r"\\"),
+        (re.compile(r"\\([\[\]\(\)\|])"), r"\1"),
+        (re.compile(r"(\d+)\\(-\d+)"), r"\1\2"),
+
+        (re.compile(r"\xff"), r"\\d+"),
+
+        # Special case: IIO ABI which a parenthesis.
+        (re.compile(r"sqrt(.*)"), r"sqrt(.*)"),
+
+        # Simplify regexes with multiple .*
+        (re.compile(r"(?:\.\*){2,}"),  ""),
+
+        # Recover dot characters
+        (re.compile(r"\xf6"), "\\."),
+        # Recover plus characters
+        (re.compile(r"\xf7"), "+"),
+    ]
+    re_has_num = re.compile(r"\\d")
+
+    # Symbol name after escape_chars that are considered a devnode basename
+    re_symbol_name =  re.compile(r"(\w|\\[\.\-\:])+$")
+
+    # List of popular group names to be skipped to minimize regex group size
+    # Use DEBUG_SUBGROUP_SIZE to detect those
+    skip_names = set(["devices", "hwmon"])
+
+    def regex_append(self, what, new):
+        """
+        Get a search group for a subset of regular expressions.
+
+        As ABI may have thousands of symbols, using a for to search all
+        regular expressions is at least O(n^2). When there are wildcards,
+        the complexity increases substantially, eventually becoming exponential.
+
+        To avoid spending too much time on them, use a logic to split
+        them into groups. The smaller the group, the better, as it would
+        mean that searches will be confined to a small number of regular
+        expressions.
+
+        The conversion to a regex subset is tricky, as we need something
+        that can be easily obtained from the sysfs symbol and from the
+        regular expression. So, we need to discard nodes that have
+        wildcards.
+
+        If it can't obtain a subgroup, place the regular expression inside
+        a special group (self.leave_others).
+        """
+
+        for search_group in reversed(new.split("/")):
+            if not search_group or search_group in self.skip_names:
+                continue
+            if self.re_symbol_name.match(search_group):
+                break
+
+        if not search_group:
+            search_group = self.leave_others
+
+        if self.debug & DEBUG_SUBGROUP_MAP:
+            self.log.debug("%s: mapped as %s", what, search_group)
+
+        try:
+            if search_group not in self.regex_group:
+                self.regex_group[search_group] = []
+
+            self.regex_group[search_group].append(re.compile(new))
+            if self.search_string:
+                if what.find(self.search_string) >= 0:
+                    print(f"What: {what}")
+        except re.PatternError:
+            self.log.warning("Ignoring '%s' as it produced an invalid regex:\n"
+                             "           '%s'", what, new)
+
+    def get_regexes(self, what):
+        """
+        Given an ABI devnode, return a list of all regular expressions that
+        may match it, based on the sub-groups created by regex_append()
+        """
+
+        re_list = []
+
+        patches = what.split("/")
+        patches.reverse()
+        patches.append(self.leave_others)
+
+        for search_group in patches:
+            if search_group in self.regex_group:
+                re_list += self.regex_group[search_group]
+
+        return re_list
+
+    def __init__(self, *args, **kwargs):
+        """
+        Override init method to get verbose argument
+        """
+
+        self.regex_group = None
+        self.search_string = None
+        self.re_string = None
+
+        if "search_string" in kwargs:
+            self.search_string = kwargs.get("search_string")
+            del kwargs["search_string"]
+
+            if self.search_string:
+
+                try:
+                    self.re_string = re.compile(self.search_string)
+                except re.PatternError as e:
+                    msg = f"{self.search_string} is not a valid regular expression"
+                    raise ValueError(msg) from e
+
+        super().__init__(*args, **kwargs)
+
+    def parse_abi(self, *args, **kwargs):
+
+        super().parse_abi(*args, **kwargs)
+
+        self.regex_group = {}
+
+        print("Converting ABI What fields into regexes...", file=sys.stderr)
+
+        for t in sorted(self.data.items(), key=lambda x: x[0]):
+            v = t[1]
+            if v.get("type") == "File":
+                continue
+
+            v["regex"] = []
+
+            for what in v.get("what", []):
+                if not what.startswith("/sys"):
+                    continue
+
+                new = what
+                for r, s in self.re_whats:
+                    try:
+                        new = r.sub(s, new)
+                    except re.PatternError as e:
+                        # Help debugging troubles with new regexes
+                        raise re.PatternError(f"{e}\nwhile re.sub('{r.pattern}', {s}, str)") from e
+
+                v["regex"].append(new)
+
+                if self.debug & DEBUG_REGEX:
+                    self.log.debug("%-90s <== %s", new, what)
+
+                # Store regex into a subgroup to speedup searches
+                self.regex_append(what, new)
+
+        if self.debug & DEBUG_SUBGROUP_DICT:
+            self.log.debug("%s", pformat(self.regex_group))
+
+        if self.debug & DEBUG_SUBGROUP_SIZE:
+            biggestd_keys = sorted(self.regex_group.keys(),
+                                   key= lambda k: len(self.regex_group[k]),
+                                   reverse=True)
+
+            print("Top regex subgroups:", file=sys.stderr)
+            for k in biggestd_keys[:10]:
+                print(f"{k} has {len(self.regex_group[k])} elements", file=sys.stderr)
+
+class SystemSymbols:
+    """Stores arguments for the class and initialize class vars"""
+
+    def graph_add_file(self, path, link=None):
+        """
+        add a file path to the sysfs graph stored at self.root
+        """
+
+        if path in self.files:
+            return
+
+        name = ""
+        ref = self.root
+        for edge in path.split("/"):
+            name += edge + "/"
+            if edge not in ref:
+                ref[edge] = {"__name": [name.rstrip("/")]}
+
+            ref = ref[edge]
+
+        if link and link not in ref["__name"]:
+            ref["__name"].append(link.rstrip("/"))
+
+        self.files.add(path)
+
+    def print_graph(self, root_prefix="", root=None, level=0):
+        """Prints a reference tree graph using UTF-8 characters"""
+
+        if not root:
+            root = self.root
+            level = 0
+
+        # Prevent endless traverse
+        if level > 5:
+            return
+
+        if level > 0:
+            prefix = "├──"
+            last_prefix = "└──"
+        else:
+            prefix = ""
+            last_prefix = ""
+
+        items = list(root.items())
+
+        names = root.get("__name", [])
+        for k, edge in items:
+            if k == "__name":
+                continue
+
+            if not k:
+                k = "/"
+
+            if len(names) > 1:
+                k += " links: " + ",".join(names[1:])
+
+            if edge == items[-1][1]:
+                print(root_prefix + last_prefix + k)
+                p = root_prefix
+                if level > 0:
+                    p += "   "
+                self.print_graph(p, edge, level + 1)
+            else:
+                print(root_prefix + prefix + k)
+                p = root_prefix + "│   "
+                self.print_graph(p, edge, level + 1)
+
+    def _walk(self, root):
+        """
+        Walk through sysfs to get all devnodes that aren't ignored.
+
+        By default, uses /sys as sysfs mounting point. If another
+        directory is used, it replaces them to /sys at the patches.
+        """
+
+        with os.scandir(root) as obj:
+            for entry in obj:
+                path = os.path.join(root, entry.name)
+                if self.sysfs:
+                    p = path.replace(self.sysfs, "/sys", count=1)
+                else:
+                    p = path
+
+                if self.re_ignore.search(p):
+                    return
+
+                # Handle link first to avoid directory recursion
+                if entry.is_symlink():
+                    real = os.path.realpath(path)
+                    if not self.sysfs:
+                        self.aliases[path] = real
+                    else:
+                        real = real.replace(self.sysfs, "/sys", count=1)
+
+                    # Add absfile location to graph if it doesn't exist
+                    if not self.re_ignore.search(real):
+                        # Add link to the graph
+                        self.graph_add_file(real, p)
+
+                elif entry.is_file():
+                    self.graph_add_file(p)
+
+                elif entry.is_dir():
+                    self._walk(path)
+
+    def __init__(self, abi, sysfs="/sys", hints=False):
+        """
+        Initialize internal variables and get a list of all files inside
+        sysfs that can currently be parsed.
+
+        Please notice that there are several entries on sysfs that aren't
+        documented as ABI. Ignore those.
+
+        The real paths will be stored under self.files. Aliases will be
+        stored in separate, as self.aliases.
+        """
+
+        self.abi = abi
+        self.log = abi.log
+
+        if sysfs != "/sys":
+            self.sysfs = sysfs.rstrip("/")
+        else:
+            self.sysfs = None
+
+        self.hints = hints
+
+        self.root = {}
+        self.aliases = {}
+        self.files = set()
+
+        dont_walk = [
+            # Those require root access and aren't documented at ABI
+            f"^{sysfs}/kernel/debug",
+            f"^{sysfs}/kernel/tracing",
+            f"^{sysfs}/fs/pstore",
+            f"^{sysfs}/fs/bpf",
+            f"^{sysfs}/fs/fuse",
+
+            # This is not documented at ABI
+            f"^{sysfs}/module",
+
+            f"^{sysfs}/fs/cgroup",  # this is big and has zero docs under ABI
+            f"^{sysfs}/firmware",   # documented elsewhere: ACPI, DT bindings
+            "sections|notes",       # aren't actually part of ABI
+
+            # kernel-parameters.txt - not easy to parse
+            "parameters",
+        ]
+
+        self.re_ignore = re.compile("|".join(dont_walk))
+
+        print(f"Reading {sysfs} directory contents...", file=sys.stderr)
+        self._walk(sysfs)
+
+    def check_file(self, refs, found):
+        """Check missing ABI symbols for a given sysfs file"""
+
+        res_list = []
+
+        try:
+            for names in refs:
+                fname = names[0]
+
+                res = {
+                    "found": False,
+                    "fname": fname,
+                    "msg": "",
+                }
+                res_list.append(res)
+
+                re_what = self.abi.get_regexes(fname)
+                if not re_what:
+                    self.abi.log.warning(f"missing rules for {fname}")
+                    continue
+
+                for name in names:
+                    for r in re_what:
+                        if self.abi.debug & DEBUG_UNDEFINED:
+                            self.log.debug("check if %s matches '%s'", name, r.pattern)
+                        if r.match(name):
+                            res["found"] = True
+                            if found:
+                                res["msg"] += f"  {fname}: regex:\n\t"
+                            continue
+
+                if self.hints and not res["found"]:
+                    res["msg"] += f"  {fname} not found. Tested regexes:\n"
+                    for r in re_what:
+                        res["msg"] += "    " + r.pattern + "\n"
+
+        except KeyboardInterrupt:
+            pass
+
+        return res_list
+
+    def _ref_interactor(self, root):
+        """Recursive function to interact over the sysfs tree"""
+
+        for k, v in root.items():
+            if isinstance(v, dict):
+                yield from self._ref_interactor(v)
+
+            if root == self.root or k == "__name":
+                continue
+
+            if self.abi.re_string:
+                fname = v["__name"][0]
+                if self.abi.re_string.search(fname):
+                    yield v
+            else:
+                yield v
+
+
+    def get_fileref(self, all_refs, chunk_size):
+        """Interactor to group refs into chunks"""
+
+        n = 0
+        refs = []
+
+        for ref in all_refs:
+            refs.append(ref)
+
+            n += 1
+            if n >= chunk_size:
+                yield refs
+                n = 0
+                refs = []
+
+        yield refs
+
+    def check_undefined_symbols(self, max_workers=None, chunk_size=50,
+                                found=None, dry_run=None):
+        """Seach ABI for sysfs symbols missing documentation"""
+
+        self.abi.parse_abi()
+
+        if self.abi.debug & DEBUG_GRAPH:
+            self.print_graph()
+
+        all_refs = []
+        for ref in self._ref_interactor(self.root):
+            all_refs.append(ref["__name"])
+
+        if dry_run:
+            print(f"Would check", file=sys.stderr)
+            for ref in all_refs:
+                print(", ".join(ref))
+
+            return
+
+        print("Starting to search symbols (it may take several minutes):",
+              file=sys.stderr)
+        start = datetime.now()
+        old_elapsed = None
+
+        # Python doesn't support multithreading due to limitations on its
+        # global lock (GIL). While Python 3.13 finally made GIL optional,
+        # there are still issues related to it. Also, we want to have
+        # backward compatibility with older versions of Python.
+        #
+        # So, use instead multiprocess. However, Python is very slow passing
+        # data from/to multiple processes. Also, it may consume lots of memory
+        # if the data to be shared is not small.  So, we need to group workload
+        # in chunks that are big enough to generate performance gains while
+        # not being so big that would cause out-of-memory.
+
+        num_refs = len(all_refs)
+        print(f"Number of references to parse: {num_refs}", file=sys.stderr)
+
+        if not max_workers:
+            max_workers = os.cpu_count()
+        elif max_workers > os.cpu_count():
+            max_workers = os.cpu_count()
+
+        max_workers = max(max_workers, 1)
+
+        max_chunk_size = int((num_refs + max_workers - 1) / max_workers)
+        chunk_size = min(chunk_size, max_chunk_size)
+        chunk_size = max(1, chunk_size)
+
+        if max_workers > 1:
+            executor = futures.ProcessPoolExecutor
+
+            # Place references in a random order. This may help improving
+            # performance, by mixing complex/simple expressions when creating
+            # chunks
+            shuffle(all_refs)
+        else:
+            # Python has a high overhead with processes. When there's just
+            # one worker, it is faster to not create a new process.
+            # Yet, User still deserves to have a progress print. So, use
+            # python's "thread", which is actually a single process, using
+            # an internal schedule to switch between tasks. No performance
+            # gains for non-IO tasks, but still it can be quickly interrupted
+            # from time to time to display progress.
+            executor = futures.ThreadPoolExecutor
+
+        not_found = []
+        f_list = []
+        with executor(max_workers=max_workers) as exe:
+            for refs in self.get_fileref(all_refs, chunk_size):
+                if refs:
+                    try:
+                        f_list.append(exe.submit(self.check_file, refs, found))
+
+                    except KeyboardInterrupt:
+                        return
+
+            total = len(f_list)
+
+            if not total:
+                if self.abi.re_string:
+                    print(f"No ABI symbol matches {self.abi.search_string}")
+                else:
+                    self.abi.log.warning("No ABI symbols found")
+                return
+
+            print(f"{len(f_list):6d} jobs queued on {max_workers} workers",
+                  file=sys.stderr)
+
+            while f_list:
+                try:
+                    t = futures.wait(f_list, timeout=1,
+                                     return_when=futures.FIRST_COMPLETED)
+
+                    done = t[0]
+
+                    for fut in done:
+                        res_list = fut.result()
+
+                        for res in res_list:
+                            if not res["found"]:
+                                not_found.append(res["fname"])
+                            if res["msg"]:
+                                print(res["msg"])
+
+                        f_list.remove(fut)
+                except KeyboardInterrupt:
+                    return
+
+                except RuntimeError as e:
+                    self.abi.log.warning(f"Future: {e}")
+                    break
+
+                if sys.stderr.isatty():
+                    elapsed = str(datetime.now() - start).split(".", maxsplit=1)[0]
+                    if len(f_list) < total:
+                        elapsed += f" ({total - len(f_list)}/{total} jobs completed).  "
+                    if elapsed != old_elapsed:
+                        print(elapsed + "\r", end="", flush=True,
+                              file=sys.stderr)
+                        old_elapsed = elapsed
+
+        elapsed = str(datetime.now() - start).split(".", maxsplit=1)[0]
+        print(elapsed, file=sys.stderr)
+
+        for f in sorted(not_found):
+            print(f"{f} not found.")
+
+
+REST_DESC = """
+Produce output in ReST format.
+
+The output is done on two sections:
+
+- Symbols: show all parsed symbols in alphabetic order;
+- Files: cross reference the content of each file with the symbols on it.
+"""
+
+
 class AbiRest:
     """Initialize an argparse subparser for rest output"""
 
@@ -725,6 +1327,71 @@ class AbiSearch:
         parser.parse_abi()
         parser.search_symbols(args.expression)
 
+UNDEFINED_DESC="""
+Check undefined ABIs on local machine.
+
+Read sysfs devnodes and check if the devnodes there are defined inside
+ABI documentation.
+
+The search logic tries to minimize the number of regular expressions to
+search per each symbol.
+
+By default, it runs on a single CPU, as Python support for CPU threads
+is still experimental, and multi-process runs on Python is very slow.
+
+On experimental tests, if the number of ABI symbols to search per devnode
+is contained on a limit of ~150 regular expressions, using a single CPU
+is a lot faster than using multiple processes. However, if the number of
+regular expressions to check is at the order of ~30000, using multiple
+CPUs speeds up the check.
+"""
+
+class AbiUndefined:
+    """
+    Initialize an argparse subparser for logic to check undefined ABI at
+    the current machine's sysfs
+    """
+
+    def __init__(self, subparsers):
+        """Initialize argparse subparsers"""
+
+        parser = subparsers.add_parser("undefined",
+                                       formatter_class=argparse.RawTextHelpFormatter,
+                                       description=UNDEFINED_DESC)
+
+        parser.add_argument("-S", "--sysfs-dir", default="/sys",
+                            help="directory where sysfs is mounted")
+        parser.add_argument("-s", "--search-string",
+                            help="search string regular expression to limit symbol search")
+        parser.add_argument("-H", "--show-hints", action="store_true",
+                            help="Hints about definitions for missing ABI symbols.")
+        parser.add_argument("-j", "--jobs", "--max-workers", type=int, default=1,
+                            help="If bigger than one, enables multiprocessing.")
+        parser.add_argument("-c", "--max-chunk-size", type=int, default=50,
+                            help="Maximum number of chunk size")
+        parser.add_argument("-f", "--found", action="store_true",
+                            help="Also show found items. "
+                                 "Helpful to debug the parser."),
+        parser.add_argument("-d", "--dry-run", action="store_true",
+                            help="Don't actually search for undefined. "
+                                 "Helpful to debug the parser."),
+
+        parser.set_defaults(func=self.run)
+
+    def run(self, args):
+        """Run subparser"""
+
+        abi = AbiRegex(args.dir, debug=args.debug,
+                       search_string=args.search_string)
+
+        abi_symbols = SystemSymbols(abi=abi, hints=args.show_hints,
+                                    sysfs=args.sysfs_dir)
+
+        abi_symbols.check_undefined_symbols(dry_run=args.dry_run,
+                                            found=args.found,
+                                            max_workers=args.jobs,
+                                            chunk_size=args.max_chunk_size)
+
 
 def main():
     """Main program"""
@@ -739,6 +1406,7 @@ def main():
     AbiRest(subparsers)
     AbiValidate(subparsers)
     AbiSearch(subparsers)
+    AbiUndefined(subparsers)
 
     args = parser.parse_args()
 
-- 
2.48.1

[RFC v2 38/38] scripts/get_abi.pl: drop now obsoleted script

[Mauro Carvalho Chehab]

As all functionalities of it were migrated to get_abi.py,
drop the now obsoleted script.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/get_abi.pl | 1103 --------------------------------------------
 1 file changed, 1103 deletions(-)
 delete mode 100755 scripts/get_abi.pl

diff --git a/scripts/get_abi.pl b/scripts/get_abi.pl
deleted file mode 100755
index de1c0354b50c..000000000000
--- a/scripts/get_abi.pl
+++ /dev/null
@@ -1,1103 +0,0 @@
-#!/usr/bin/env perl
-# SPDX-License-Identifier: GPL-2.0
-
-BEGIN { $Pod::Usage::Formatter = 'Pod::Text::Termcap'; }
-
-use strict;
-use warnings;
-use utf8;
-use Pod::Usage qw(pod2usage);
-use Getopt::Long;
-use File::Find;
-use IO::Handle;
-use Fcntl ':mode';
-use Cwd 'abs_path';
-use Data::Dumper;
-
-my $help = 0;
-my $hint = 0;
-my $man = 0;
-my $debug = 0;
-my $enable_lineno = 0;
-my $show_warnings = 1;
-my $prefix="Documentation/ABI";
-my $sysfs_prefix="/sys";
-my $search_string;
-
-# Debug options
-my $dbg_what_parsing = 1;
-my $dbg_what_open = 2;
-my $dbg_dump_abi_structs = 4;
-my $dbg_undefined = 8;
-
-$Data::Dumper::Indent = 1;
-$Data::Dumper::Terse = 1;
-
-#
-# If true, assumes that the description is formatted with ReST
-#
-my $description_is_rst = 1;
-
-GetOptions(
-	"debug=i" => \$debug,
-	"enable-lineno" => \$enable_lineno,
-	"rst-source!" => \$description_is_rst,
-	"dir=s" => \$prefix,
-	'help|?' => \$help,
-	"show-hints" => \$hint,
-	"search-string=s" => \$search_string,
-	man => \$man
-) or pod2usage(2);
-
-pod2usage(1) if $help;
-pod2usage(-exitstatus => 0, -noperldoc, -verbose => 2) if $man;
-
-pod2usage(2) if (scalar @ARGV < 1 || @ARGV > 2);
-
-my ($cmd, $arg) = @ARGV;
-
-pod2usage(2) if ($cmd ne "search" && $cmd ne "rest" && $cmd ne "validate" && $cmd ne "undefined");
-pod2usage(2) if ($cmd eq "search" && !$arg);
-
-require Data::Dumper if ($debug & $dbg_dump_abi_structs);
-
-my %data;
-my %symbols;
-
-#
-# Displays an error message, printing file name and line
-#
-sub parse_error($$$$) {
-	my ($file, $ln, $msg, $data) = @_;
-
-	return if (!$show_warnings);
-
-	$data =~ s/\s+$/\n/;
-
-	print STDERR "Warning: file $file#$ln:\n\t$msg";
-
-	if ($data ne "") {
-		print STDERR ". Line\n\t\t$data";
-	} else {
-	    print STDERR "\n";
-	}
-}
-
-#
-# Parse an ABI file, storing its contents at %data
-#
-sub parse_abi {
-	my $file = $File::Find::name;
-
-	my $mode = (stat($file))[2];
-	return if ($mode & S_IFDIR);
-	return if ($file =~ m,/README,);
-	return if ($file =~ m,/\.,);
-	return if ($file =~ m,\.(rej|org|orig|bak)$,);
-
-	my $name = $file;
-	$name =~ s,.*/,,;
-
-	my $fn = $file;
-	$fn =~ s,.*Documentation/ABI/,,;
-
-	my $nametag = "File $fn";
-	$data{$nametag}->{what} = "File $name";
-	$data{$nametag}->{type} = "File";
-	$data{$nametag}->{file} = $name;
-	$data{$nametag}->{filepath} = $file;
-	$data{$nametag}->{is_file} = 1;
-	$data{$nametag}->{line_no} = 1;
-
-	my $type = $file;
-	$type =~ s,.*/(.*)/.*,$1,;
-
-	my $what;
-	my $new_what;
-	my $tag = "";
-	my $ln;
-	my $xrefs;
-	my $space;
-	my @labels;
-	my $label = "";
-
-	print STDERR "Opening $file\n" if ($debug & $dbg_what_open);
-	open IN, $file;
-	while(<IN>) {
-		$ln++;
-		if (m/^(\S+)(:\s*)(.*)/i) {
-			my $new_tag = lc($1);
-			my $sep = $2;
-			my $content = $3;
-
-			if (!($new_tag =~ m/(what|where|date|kernelversion|contact|description|users)/)) {
-				if ($tag eq "description") {
-					# New "tag" is actually part of
-					# description. Don't consider it a tag
-					$new_tag = "";
-				} elsif ($tag ne "") {
-					parse_error($file, $ln, "tag '$tag' is invalid", $_);
-				}
-			}
-
-			# Invalid, but it is a common mistake
-			if ($new_tag eq "where") {
-				parse_error($file, $ln, "tag 'Where' is invalid. Should be 'What:' instead", "");
-				$new_tag = "what";
-			}
-
-			if ($new_tag =~ m/what/) {
-				$space = "";
-				$content =~ s/[,.;]$//;
-
-				push @{$symbols{$content}->{file}}, " $file:" . ($ln - 1);
-
-				if ($tag =~ m/what/) {
-					$what .= "\xac" . $content;
-				} else {
-					if ($what) {
-						parse_error($file, $ln, "What '$what' doesn't have a description", "") if (!$data{$what}->{description});
-
-						foreach my $w(split /\xac/, $what) {
-							$symbols{$w}->{xref} = $what;
-						};
-					}
-
-					$what = $content;
-					$label = $content;
-					$new_what = 1;
-				}
-				push @labels, [($content, $label)];
-				$tag = $new_tag;
-
-				push @{$data{$nametag}->{symbols}}, $content if ($data{$nametag}->{what});
-				next;
-			}
-
-			if ($tag ne "" && $new_tag) {
-				$tag = $new_tag;
-
-				if ($new_what) {
-					@{$data{$what}->{label_list}} = @labels if ($data{$nametag}->{what});
-					@labels = ();
-					$label = "";
-					$new_what = 0;
-
-					$data{$what}->{type} = $type;
-					if (!defined($data{$what}->{file})) {
-						$data{$what}->{file} = $name;
-						$data{$what}->{filepath} = $file;
-					} else {
-						$data{$what}->{description} .= "\n\n" if (defined($data{$what}->{description}));
-						if ($name ne $data{$what}->{file}) {
-							$data{$what}->{file} .= " " . $name;
-							$data{$what}->{filepath} .= " " . $file;
-						}
-					}
-					print STDERR "\twhat: $what\n" if ($debug & $dbg_what_parsing);
-					$data{$what}->{line_no} = $ln;
-				} else {
-					$data{$what}->{line_no} = $ln if (!defined($data{$what}->{line_no}));
-				}
-
-				if (!$what) {
-					parse_error($file, $ln, "'What:' should come first:", $_);
-					next;
-				}
-				if ($new_tag eq "description") {
-					$sep =~ s,:, ,;
-					$content = ' ' x length($new_tag) . $sep . $content;
-					while ($content =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
-					if ($content =~ m/^(\s*)(\S.*)$/) {
-						# Preserve initial spaces for the first line
-						$space = $1;
-						$content = "$2\n";
-						$data{$what}->{$tag} .= $content;
-					} else {
-						undef($space);
-					}
-
-				} else {
-					$data{$what}->{$tag} = $content;
-				}
-				next;
-			}
-		}
-
-		# Store any contents before tags at the database
-		if (!$tag && $data{$nametag}->{what}) {
-			$data{$nametag}->{description} .= $_;
-			next;
-		}
-
-		if ($tag eq "description") {
-			my $content = $_;
-			while ($content =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {}
-			if (m/^\s*\n/) {
-				$data{$what}->{$tag} .= "\n";
-				next;
-			}
-
-			if (!defined($space)) {
-				# Preserve initial spaces for the first line
-				if ($content =~ m/^(\s*)(\S.*)$/) {
-					$space = $1;
-					$content = "$2\n";
-				}
-			} else {
-				$space = "" if (!($content =~ s/^($space)//));
-			}
-			$data{$what}->{$tag} .= $content;
-
-			next;
-		}
-		if (m/^\s*(.*)/) {
-			$data{$what}->{$tag} .= "\n$1";
-			$data{$what}->{$tag} =~ s/\n+$//;
-			next;
-		}
-
-		# Everything else is error
-		parse_error($file, $ln, "Unexpected content", $_);
-	}
-	$data{$nametag}->{description} =~ s/^\n+// if ($data{$nametag}->{description});
-	if ($what) {
-		parse_error($file, $ln, "What '$what' doesn't have a description", "") if (!$data{$what}->{description});
-
-		foreach my $w(split /\xac/,$what) {
-			$symbols{$w}->{xref} = $what;
-		};
-	}
-	close IN;
-}
-
-sub create_labels {
-	my %labels;
-
-	foreach my $what (keys %data) {
-		next if ($data{$what}->{file} eq "File");
-
-		foreach my $p (@{$data{$what}->{label_list}}) {
-			my ($content, $label) = @{$p};
-			$label = "abi_" . $label . " ";
-			$label =~ tr/A-Z/a-z/;
-
-			# Convert special chars to "_"
-			$label =~s/([\x00-\x2f\x3a-\x40\x5b-\x60\x7b-\xff])/_/g;
-			$label =~ s,_+,_,g;
-			$label =~ s,_$,,;
-
-			# Avoid duplicated labels
-			while (defined($labels{$label})) {
-			    my @chars = ("A".."Z", "a".."z");
-			    $label .= $chars[rand @chars];
-			}
-			$labels{$label} = 1;
-
-			$data{$what}->{label} = $label;
-
-			# only one label is enough
-			last;
-		}
-	}
-}
-
-#
-# Outputs the book on ReST format
-#
-
-# \b doesn't work well with paths. So, we need to define something else:
-# Boundaries are punct characters, spaces and end-of-line
-my $start = qr {(^|\s|\() }x;
-my $bondary = qr { ([,.:;\)\s]|\z) }x;
-my $xref_match = qr { $start(\/(sys|config|proc|dev|kvd)\/[^,.:;\)\s]+)$bondary }x;
-my $symbols = qr { ([\x01-\x08\x0e-\x1f\x21-\x2f\x3a-\x40\x7b-\xff]) }x;
-
-sub output_rest {
-	create_labels();
-
-	my $part = "";
-
-	foreach my $what (sort {
-				($data{$a}->{type} eq "File") cmp ($data{$b}->{type} eq "File") ||
-				$a cmp $b
-			       } keys %data) {
-		my $type = $data{$what}->{type};
-
-		my @file = split / /, $data{$what}->{file};
-		my @filepath = split / /, $data{$what}->{filepath};
-
-		if ($enable_lineno) {
-			printf ".. LINENO %s%s#%s\n\n",
-			       $prefix, $file[0],
-			       $data{$what}->{line_no};
-		}
-
-		my $w = $what;
-
-		if ($type ne "File") {
-			my $cur_part = $what;
-			if ($what =~ '/') {
-				if ($what =~ m#^(\/?(?:[\w\-]+\/?){1,2})#) {
-					$cur_part = "Symbols under $1";
-					$cur_part =~ s,/$,,;
-				}
-			}
-
-			if ($cur_part ne "" && $part ne $cur_part) {
-			    $part = $cur_part;
-			    my $bar = $part;
-			    $bar =~ s/./-/g;
-			    print "$part\n$bar\n\n";
-			}
-
-			printf ".. _%s:\n\n", $data{$what}->{label};
-
-			my @names = split /\xac/,$w;
-			my $len = 0;
-
-			foreach my $name (@names) {
-				$name =~ s/$symbols/\\$1/g;
-				$name = "**$name**";
-				$len = length($name) if (length($name) > $len);
-			}
-
-			print "+-" . "-" x $len . "-+\n";
-			foreach my $name (@names) {
-				printf "| %s", $name . " " x ($len - length($name)) . " |\n";
-				print "+-" . "-" x $len . "-+\n";
-			}
-
-			print "\n";
-		}
-
-		for (my $i = 0; $i < scalar(@filepath); $i++) {
-			my $path = $filepath[$i];
-			my $f = $file[$i];
-
-			$path =~ s,.*/(.*/.*),$1,;;
-			$path =~ s,[/\-],_,g;;
-			my $fileref = "abi_file_".$path;
-
-			if ($type eq "File") {
-				print ".. _$fileref:\n\n";
-			} else {
-				print "Defined on file :ref:`$f <$fileref>`\n\n";
-			}
-		}
-
-		if ($type eq "File") {
-			my $bar = $w;
-			$bar =~ s/./-/g;
-			print "$w\n$bar\n\n";
-		}
-
-		my $desc = "";
-		$desc = $data{$what}->{description} if (defined($data{$what}->{description}));
-		$desc =~ s/\s+$/\n/;
-
-		if (!($desc =~ /^\s*$/)) {
-			if ($description_is_rst) {
-				# Remove title markups from the description
-				# Having titles inside ABI files will only work if extra
-				# care would be taken in order to strictly follow the same
-				# level order for each markup.
-				$desc =~ s/\n[\-\*\=\^\~]+\n/\n\n/g;
-
-				# Enrich text by creating cross-references
-
-				my $new_desc = "";
-				my $init_indent = -1;
-				my $literal_indent = -1;
-
-				open(my $fh, "+<", \$desc);
-				while (my $d = <$fh>) {
-					my $indent = $d =~ m/^(\s+)/;
-					my $spaces = length($indent);
-					$init_indent = $indent if ($init_indent < 0);
-					if ($literal_indent >= 0) {
-						if ($spaces > $literal_indent) {
-							$new_desc .= $d;
-							next;
-						} else {
-							$literal_indent = -1;
-						}
-					} else {
-						if ($d =~ /()::$/ && !($d =~ /^\s*\.\./)) {
-							$literal_indent = $spaces;
-						}
-					}
-
-					$d =~ s,Documentation/(?!devicetree)(\S+)\.rst,:doc:`/$1`,g;
-
-					my @matches = $d =~ m,Documentation/ABI/([\w\/\-]+),g;
-					foreach my $f (@matches) {
-						my $xref = $f;
-						my $path = $f;
-						$path =~ s,.*/(.*/.*),$1,;;
-						$path =~ s,[/\-],_,g;;
-						$xref .= " <abi_file_" . $path . ">";
-						$d =~ s,\bDocumentation/ABI/$f\b,:ref:`$xref`,g;
-					}
-
-					# Seek for cross reference symbols like /sys/...
-					@matches = $d =~ m/$xref_match/g;
-
-					foreach my $s (@matches) {
-						next if (!($s =~ m,/,));
-						if (defined($data{$s}) && defined($data{$s}->{label})) {
-							my $xref = $s;
-
-							$xref =~ s/$symbols/\\$1/g;
-							$xref = ":ref:`$xref <" . $data{$s}->{label} . ">`";
-
-							$d =~ s,$start$s$bondary,$1$xref$2,g;
-						}
-					}
-					$new_desc .= $d;
-				}
-				close $fh;
-
-
-				print "$new_desc\n\n";
-			} else {
-				$desc =~ s/^\s+//;
-
-				# Remove title markups from the description, as they won't work
-				$desc =~ s/\n[\-\*\=\^\~]+\n/\n\n/g;
-
-				if ($desc =~ m/\:\n/ || $desc =~ m/\n[\t ]+/  || $desc =~ m/[\x00-\x08\x0b-\x1f\x7b-\xff]/) {
-					# put everything inside a code block
-					$desc =~ s/\n/\n /g;
-
-					print "::\n\n";
-					print " $desc\n\n";
-				} else {
-					# Escape any special chars from description
-					$desc =~s/([\x00-\x08\x0b-\x1f\x21-\x2a\x2d\x2f\x3c-\x40\x5c\x5e-\x60\x7b-\xff])/\\$1/g;
-					print "$desc\n\n";
-				}
-			}
-		} else {
-			print "DESCRIPTION MISSING for $what\n\n" if (!$data{$what}->{is_file});
-		}
-
-		if ($data{$what}->{symbols}) {
-			printf "Has the following ABI:\n\n";
-
-			foreach my $content(@{$data{$what}->{symbols}}) {
-				my $label = $data{$symbols{$content}->{xref}}->{label};
-
-				# Escape special chars from content
-				$content =~s/([\x00-\x1f\x21-\x2f\x3a-\x40\x7b-\xff])/\\$1/g;
-
-				print "- :ref:`$content <$label>`\n\n";
-			}
-		}
-
-		if (defined($data{$what}->{users})) {
-			my $users = $data{$what}->{users};
-
-			$users =~ s/\n/\n\t/g;
-			printf "Users:\n\t%s\n\n", $users if ($users ne "");
-		}
-
-	}
-}
-
-#
-# Searches for ABI symbols
-#
-sub search_symbols {
-	foreach my $what (sort keys %data) {
-		next if (!($what =~ m/($arg)/));
-
-		my $type = $data{$what}->{type};
-		next if ($type eq "File");
-
-		my $file = $data{$what}->{filepath};
-
-		$what =~ s/\xac/, /g;
-		my $bar = $what;
-		$bar =~ s/./-/g;
-
-		print "\n$what\n$bar\n\n";
-
-		my $kernelversion = $data{$what}->{kernelversion} if (defined($data{$what}->{kernelversion}));
-		my $contact = $data{$what}->{contact} if (defined($data{$what}->{contact}));
-		my $users = $data{$what}->{users} if (defined($data{$what}->{users}));
-		my $date = $data{$what}->{date} if (defined($data{$what}->{date}));
-		my $desc = $data{$what}->{description} if (defined($data{$what}->{description}));
-
-		$kernelversion =~ s/^\s+// if ($kernelversion);
-		$contact =~ s/^\s+// if ($contact);
-		if ($users) {
-			$users =~ s/^\s+//;
-			$users =~ s/\n//g;
-		}
-		$date =~ s/^\s+// if ($date);
-		$desc =~ s/^\s+// if ($desc);
-
-		printf "Kernel version:\t\t%s\n", $kernelversion if ($kernelversion);
-		printf "Date:\t\t\t%s\n", $date if ($date);
-		printf "Contact:\t\t%s\n", $contact if ($contact);
-		printf "Users:\t\t\t%s\n", $users if ($users);
-		print "Defined on file(s):\t$file\n\n";
-		print "Description:\n\n$desc";
-	}
-}
-
-# Exclude /sys/kernel/debug and /sys/kernel/tracing from the search path
-sub dont_parse_special_attributes {
-	if (($File::Find::dir =~ m,^/sys/kernel,)) {
-		return grep {!/(debug|tracing)/ } @_;
-	}
-
-	if (($File::Find::dir =~ m,^/sys/fs,)) {
-		return grep {!/(pstore|bpf|fuse)/ } @_;
-	}
-
-	return @_
-}
-
-my %leaf;
-my %aliases;
-my @files;
-my %root;
-
-sub graph_add_file {
-	my $file = shift;
-	my $type = shift;
-
-	my $dir = $file;
-	$dir =~ s,^(.*/).*,$1,;
-	$file =~ s,.*/,,;
-
-	my $name;
-	my $file_ref = \%root;
-	foreach my $edge(split "/", $dir) {
-		$name .= "$edge/";
-		if (!defined ${$file_ref}{$edge}) {
-			${$file_ref}{$edge} = { };
-		}
-		$file_ref = \%{$$file_ref{$edge}};
-		${$file_ref}{"__name"} = [ $name ];
-	}
-	$name .= "$file";
-	${$file_ref}{$file} = {
-		"__name" => [ $name ]
-	};
-
-	return \%{$$file_ref{$file}};
-}
-
-sub graph_add_link {
-	my $file = shift;
-	my $link = shift;
-
-	# Traverse graph to find the reference
-	my $file_ref = \%root;
-	foreach my $edge(split "/", $file) {
-		$file_ref = \%{$$file_ref{$edge}} || die "Missing node!";
-	}
-
-	# do a BFS
-
-	my @queue;
-	my %seen;
-	my $st;
-
-	push @queue, $file_ref;
-	$seen{$start}++;
-
-	while (@queue) {
-		my $v = shift @queue;
-		my @child = keys(%{$v});
-
-		foreach my $c(@child) {
-			next if $seen{$$v{$c}};
-			next if ($c eq "__name");
-
-			if (!defined($$v{$c}{"__name"})) {
-				printf STDERR "Error: Couldn't find a non-empty name on a children of $file/.*: ";
-				print STDERR Dumper(%{$v});
-				exit;
-			}
-
-			# Add new name
-			my $name = @{$$v{$c}{"__name"}}[0];
-			if ($name =~ s#^$file/#$link/#) {
-				push @{$$v{$c}{"__name"}}, $name;
-			}
-			# Add child to the queue and mark as seen
-			push @queue, $$v{$c};
-			$seen{$c}++;
-		}
-	}
-}
-
-my $escape_symbols = qr { ([\x01-\x08\x0e-\x1f\x21-\x29\x2b-\x2d\x3a-\x40\x7b-\xfe]) }x;
-sub parse_existing_sysfs {
-	my $file = $File::Find::name;
-
-	my $mode = (lstat($file))[2];
-	my $abs_file = abs_path($file);
-
-	my @tmp;
-	push @tmp, $file;
-	push @tmp, $abs_file if ($abs_file ne $file);
-
-	foreach my $f(@tmp) {
-		# Ignore cgroup, as this is big and has zero docs under ABI
-		return if ($f =~ m#^/sys/fs/cgroup/#);
-
-		# Ignore firmware as it is documented elsewhere
-		# Either ACPI or under Documentation/devicetree/bindings/
-		return if ($f =~ m#^/sys/firmware/#);
-
-		# Ignore some sysfs nodes that aren't actually part of ABI
-		return if ($f =~ m#/sections|notes/#);
-
-		# Would need to check at
-		# Documentation/admin-guide/kernel-parameters.txt, but this
-		# is not easily parseable.
-		return if ($f =~ m#/parameters/#);
-	}
-
-	if (S_ISLNK($mode)) {
-		$aliases{$file} = $abs_file;
-		return;
-	}
-
-	return if (S_ISDIR($mode));
-
-	# Trivial: file is defined exactly the same way at ABI What:
-	return if (defined($data{$file}));
-	return if (defined($data{$abs_file}));
-
-	push @files, graph_add_file($abs_file, "file");
-}
-
-sub get_leave($)
-{
-	my $what = shift;
-	my $leave;
-
-	my $l = $what;
-	my $stop = 1;
-
-	$leave = $l;
-	$leave =~ s,/$,,;
-	$leave =~ s,.*/,,;
-	$leave =~ s/[\(\)]//g;
-
-	# $leave is used to improve search performance at
-	# check_undefined_symbols, as the algorithm there can seek
-	# for a small number of "what". It also allows giving a
-	# hint about a leave with the same name somewhere else.
-	# However, there are a few occurences where the leave is
-	# either a wildcard or a number. Just group such cases
-	# altogether.
-	if ($leave =~ m/\.\*/ || $leave eq "" || $leave =~ /\\d/) {
-		$leave = "others";
-	}
-
-	return $leave;
-}
-
-my @not_found;
-
-sub check_file($$)
-{
-	my $file_ref = shift;
-	my $names_ref = shift;
-	my @names = @{$names_ref};
-	my $file = $names[0];
-
-	my $found_string;
-
-	my $leave = get_leave($file);
-	if (!defined($leaf{$leave})) {
-		$leave = "others";
-	}
-	my @expr = @{$leaf{$leave}->{expr}};
-	die ("\rmissing rules for $leave") if (!defined($leaf{$leave}));
-
-	my $path = $file;
-	$path =~ s,(.*/).*,$1,;
-
-	if ($search_string) {
-		return if (!($file =~ m#$search_string#));
-		$found_string = 1;
-	}
-
-	for (my $i = 0; $i < @names; $i++) {
-		if ($found_string && $hint) {
-			if (!$i) {
-				print STDERR "--> $names[$i]\n";
-			} else {
-				print STDERR "    $names[$i]\n";
-			}
-		}
-		foreach my $re (@expr) {
-			print STDERR "$names[$i] =~ /^$re\$/\n" if ($debug && $dbg_undefined);
-			if ($names[$i] =~ $re) {
-				return;
-			}
-		}
-	}
-
-	if ($leave ne "others") {
-		my @expr = @{$leaf{"others"}->{expr}};
-		for (my $i = 0; $i < @names; $i++) {
-			foreach my $re (@expr) {
-				print STDERR "$names[$i] =~ /^$re\$/\n" if ($debug && $dbg_undefined);
-				if ($names[$i] =~ $re) {
-					return;
-				}
-			}
-		}
-	}
-
-	push @not_found, $file if (!$search_string || $found_string);
-
-	if ($hint && (!$search_string || $found_string)) {
-		my $what = $leaf{$leave}->{what};
-		$what =~ s/\xac/\n\t/g;
-		if ($leave ne "others") {
-			print STDERR "\r    more likely regexes:\n\t$what\n";
-		} else {
-			print STDERR "\r    tested regexes:\n\t$what\n";
-		}
-	}
-}
-
-sub check_undefined_symbols {
-	my $num_files = scalar @files;
-	my $next_i = 0;
-	my $start_time = times;
-
-	@files = sort @files;
-
-	my $last_time = $start_time;
-
-	# When either debug or hint is enabled, there's no sense showing
-	# progress, as the progress will be overriden.
-	if ($hint || ($debug && $dbg_undefined)) {
-		$next_i = $num_files;
-	}
-
-	my $is_console;
-	$is_console = 1 if (-t STDERR);
-
-	for (my $i = 0; $i < $num_files; $i++) {
-		my $file_ref = $files[$i];
-		my @names = @{$$file_ref{"__name"}};
-
-		check_file($file_ref, \@names);
-
-		my $cur_time = times;
-
-		if ($i == $next_i || $cur_time > $last_time + 1) {
-			my $percent = $i * 100 / $num_files;
-
-			my $tm = $cur_time - $start_time;
-			my $time = sprintf "%d:%02d", int($tm), 60 * ($tm - int($tm));
-
-			printf STDERR "\33[2K\r", if ($is_console);
-			printf STDERR "%s: processing sysfs files... %i%%: $names[0]", $time, $percent;
-			printf STDERR "\n", if (!$is_console);
-			STDERR->flush();
-
-			$next_i = int (($percent + 1) * $num_files / 100);
-			$last_time = $cur_time;
-		}
-	}
-
-	my $cur_time = times;
-	my $tm = $cur_time - $start_time;
-	my $time = sprintf "%d:%02d", int($tm), 60 * ($tm - int($tm));
-
-	printf STDERR "\33[2K\r", if ($is_console);
-	printf STDERR "%s: processing sysfs files... done\n", $time;
-
-	foreach my $file (@not_found) {
-		print "$file not found.\n";
-	}
-}
-
-sub undefined_symbols {
-	print STDERR "Reading $sysfs_prefix directory contents...";
-	find({
-		wanted =>\&parse_existing_sysfs,
-		preprocess =>\&dont_parse_special_attributes,
-		no_chdir => 1
-	     }, $sysfs_prefix);
-	print STDERR "done.\n";
-
-	$leaf{"others"}->{what} = "";
-
-	print STDERR "Converting ABI What fields into regexes...";
-	foreach my $w (sort keys %data) {
-		foreach my $what (split /\xac/,$w) {
-			next if (!($what =~ m/^$sysfs_prefix/));
-
-			# Convert what into regular expressions
-
-			# Escape dot characters
-			$what =~ s/\./\xf6/g;
-
-			# Temporarily change [0-9]+ type of patterns
-			$what =~ s/\[0\-9\]\+/\xff/g;
-
-			# Temporarily change [\d+-\d+] type of patterns
-			$what =~ s/\[0\-\d+\]/\xff/g;
-			$what =~ s/\[(\d+)\]/\xf4$1\xf5/g;
-
-			# Temporarily change [0-9] type of patterns
-			$what =~ s/\[(\d)\-(\d)\]/\xf4$1-$2\xf5/g;
-
-			# Handle multiple option patterns
-			$what =~ s/[\{\<\[]([\w_]+)(?:[,|]+([\w_]+)){1,}[\}\>\]]/($1|$2)/g;
-
-			# Handle wildcards
-			$what =~ s,\*,.*,g;
-			$what =~ s,/\xf6..,/.*,g;
-			$what =~ s/\<[^\>]+\>/.*/g;
-			$what =~ s/\{[^\}]+\}/.*/g;
-			$what =~ s/\[[^\]]+\]/.*/g;
-
-			$what =~ s/[XYZ]/.*/g;
-
-			# Recover [0-9] type of patterns
-			$what =~ s/\xf4/[/g;
-			$what =~ s/\xf5/]/g;
-
-			# Remove duplicated spaces
-			$what =~ s/\s+/ /g;
-
-			# Special case: this ABI has a parenthesis on it
-			$what =~ s/sqrt\(x^2\+y^2\+z^2\)/sqrt\(x^2\+y^2\+z^2\)/;
-
-			# Special case: drop comparition as in:
-			#	What: foo = <something>
-			# (this happens on a few IIO definitions)
-			$what =~ s,\s*\=.*$,,;
-
-			# Escape all other symbols
-			$what =~ s/$escape_symbols/\\$1/g;
-			$what =~ s/\\\\/\\/g;
-			$what =~ s/\\([\[\]\(\)\|])/$1/g;
-			$what =~ s/(\d+)\\(-\d+)/$1$2/g;
-
-			$what =~ s/\xff/\\d+/g;
-
-			# Special case: IIO ABI which a parenthesis.
-			$what =~ s/sqrt(.*)/sqrt\(.*\)/;
-
-			# Simplify regexes with multiple .*
-			$what =~ s#(?:\.\*){2,}##g;
-#			$what =~ s#\.\*/\.\*#.*#g;
-
-			# Recover dot characters
-			$what =~ s/\xf6/\./g;
-
-			my $leave = get_leave($what);
-
-			my $added = 0;
-			foreach my $l (split /\|/, $leave) {
-				if (defined($leaf{$l})) {
-					next if ($leaf{$l}->{what} =~ m/\b$what\b/);
-					$leaf{$l}->{what} .= "\xac" . $what;
-					$added = 1;
-				} else {
-					$leaf{$l}->{what} = $what;
-					$added = 1;
-				}
-			}
-			if ($search_string && $added) {
-				print STDERR "What: $what\n" if ($what =~ m#$search_string#);
-			}
-
-		}
-	}
-	# Compile regexes
-	foreach my $l (sort keys %leaf) {
-		my @expr;
-		foreach my $w(sort split /\xac/, $leaf{$l}->{what}) {
-			push @expr, qr /^$w$/;
-		}
-		$leaf{$l}->{expr} = \@expr;
-	}
-
-	# Take links into account
-	foreach my $link (sort keys %aliases) {
-		my $abs_file = $aliases{$link};
-		graph_add_link($abs_file, $link);
-	}
-	print STDERR "done.\n";
-
-	check_undefined_symbols;
-}
-
-# Ensure that the prefix will always end with a slash
-# While this is not needed for find, it makes the patch nicer
-# with --enable-lineno
-$prefix =~ s,/?$,/,;
-
-if ($cmd eq "undefined" || $cmd eq "search") {
-	$show_warnings = 0;
-}
-#
-# Parses all ABI files located at $prefix dir
-#
-find({wanted =>\&parse_abi, no_chdir => 1}, $prefix);
-
-print STDERR Data::Dumper->Dump([\%data], [qw(*data)]) if ($debug & $dbg_dump_abi_structs);
-
-#
-# Handles the command
-#
-if ($cmd eq "undefined") {
-	undefined_symbols;
-} elsif ($cmd eq "search") {
-	search_symbols;
-} else {
-	if ($cmd eq "rest") {
-		output_rest;
-	}
-
-	# Warn about duplicated ABI entries
-	foreach my $what(sort keys %symbols) {
-		my @files = @{$symbols{$what}->{file}};
-
-		next if (scalar(@files) == 1);
-
-		printf STDERR "Warning: $what is defined %d times: @files\n",
-		    scalar(@files);
-	}
-}
-
-__END__
-
-=head1 NAME
-
-get_abi.pl - parse the Linux ABI files and produce a ReST book.
-
-=head1 SYNOPSIS
-
-B<get_abi.pl> [--debug <level>] [--enable-lineno] [--man] [--help]
-	       [--(no-)rst-source] [--dir=<dir>] [--show-hints]
-	       [--search-string <regex>]
-	       <COMMAND> [<ARGUMENT>]
-
-Where B<COMMAND> can be:
-
-=over 8
-
-B<search> I<SEARCH_REGEX> - search for I<SEARCH_REGEX> inside ABI
-
-B<rest>                   - output the ABI in ReST markup language
-
-B<validate>               - validate the ABI contents
-
-B<undefined>              - existing symbols at the system that aren't
-                            defined at Documentation/ABI
-
-=back
-
-=head1 OPTIONS
-
-=over 8
-
-=item B<--dir>
-
-Changes the location of the ABI search. By default, it uses
-the Documentation/ABI directory.
-
-=item B<--rst-source> and B<--no-rst-source>
-
-The input file may be using ReST syntax or not. Those two options allow
-selecting between a rst-compliant source ABI (B<--rst-source>), or a
-plain text that may be violating ReST spec, so it requres some escaping
-logic (B<--no-rst-source>).
-
-=item B<--enable-lineno>
-
-Enable output of .. LINENO lines.
-
-=item B<--debug> I<debug level>
-
-Print debug information according with the level, which is given by the
-following bitmask:
-
-    -  1: Debug parsing What entries from ABI files;
-    -  2: Shows what files are opened from ABI files;
-    -  4: Dump the structs used to store the contents of the ABI files.
-
-=item B<--show-hints>
-
-Show hints about possible definitions for the missing ABI symbols.
-Used only when B<undefined>.
-
-=item B<--search-string> I<regex string>
-
-Show only occurences that match a search string.
-Used only when B<undefined>.
-
-=item B<--help>
-
-Prints a brief help message and exits.
-
-=item B<--man>
-
-Prints the manual page and exits.
-
-=back
-
-=head1 DESCRIPTION
-
-Parse the Linux ABI files from ABI DIR (usually located at Documentation/ABI),
-allowing to search for ABI symbols or to produce a ReST book containing
-the Linux ABI documentation.
-
-=head1 EXAMPLES
-
-Search for all stable symbols with the word "usb":
-
-=over 8
-
-$ scripts/get_abi.pl search usb --dir Documentation/ABI/stable
-
-=back
-
-Search for all symbols that match the regex expression "usb.*cap":
-
-=over 8
-
-$ scripts/get_abi.pl search usb.*cap
-
-=back
-
-Output all obsoleted symbols in ReST format
-
-=over 8
-
-$ scripts/get_abi.pl rest --dir Documentation/ABI/obsolete
-
-=back
-
-=head1 BUGS
-
-Report bugs to Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
-
-=head1 COPYRIGHT
-
-Copyright (c) 2016-2021 by Mauro Carvalho Chehab <mchehab+huawei@kernel.org>.
-
-License GPLv2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>.
-
-This is free software: you are free to change and redistribute it.
-There is NO WARRANTY, to the extent permitted by law.
-
-=cut
-- 
2.48.1

Re: [RFC v2 29/38] docs: thunderbolt: Allow creating cross-references for ABI

[Mika Westerberg]

On Tue, Jan 28, 2025 at 01:06:18AM +0100, Mauro Carvalho Chehab wrote:
> Now that Documentation/ABI is processed by automarkup, let it
> generate cross-references for the corresponding ABI file.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

Looks good to me,

Acked-by: Mika Westerberg <mika.westerberg@linux.intel.com>

Re: [RFC v2 32/38] docs: iio: Allow creating cross-references ABI

[Jonathan Cameron]

On Tue, 28 Jan 2025 01:06:21 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> Now that Documentation/ABI is processed by automarkup, let it
> generate cross-references for the corresponding ABI file.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Acked-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> ---
>  Documentation/driver-api/iio/core.rst | 2 +-
>  Documentation/iio/iio_devbuf.rst      | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/driver-api/iio/core.rst b/Documentation/driver-api/iio/core.rst
> index dfe438dc91a7..42b580fb2989 100644
> --- a/Documentation/driver-api/iio/core.rst
> +++ b/Documentation/driver-api/iio/core.rst
> @@ -60,7 +60,7 @@ directory.  Common attributes are:
>  * :file:`sampling_frequency_available`, available discrete set of sampling
>    frequency values for device.
>  * Available standard attributes for IIO devices are described in the
> -  :file:`Documentation/ABI/testing/sysfs-bus-iio` file in the Linux kernel
> +  :file:Documentation/ABI/testing/sysfs-bus-iio file in the Linux kernel
>    sources.
>  
>  IIO device channels
> diff --git a/Documentation/iio/iio_devbuf.rst b/Documentation/iio/iio_devbuf.rst
> index 9919e4792d0e..dca1f0200b0d 100644
> --- a/Documentation/iio/iio_devbuf.rst
> +++ b/Documentation/iio/iio_devbuf.rst
> @@ -148,5 +148,5 @@ applied), however there are corner cases in which the buffered data may be found
>  in a processed form. Please note that these corner cases are not addressed by
>  this documentation.
>  
> -Please see ``Documentation/ABI/testing/sysfs-bus-iio`` for a complete
> +Please see Documentation/ABI/testing/sysfs-bus-iio for a complete
>  description of the attributes.

Re: [RFC v2 01/38] docs: power: video.rst: fix a footnote reference

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Footnotes require an space (or a "\ ") before them, as otherwise
> Sphinx/Docutils won't understand it.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
>  Documentation/power/video.rst | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/Documentation/power/video.rst b/Documentation/power/video.rst
> index 337a2ba9f32f..8ab2458d1304 100644
> --- a/Documentation/power/video.rst
> +++ b/Documentation/power/video.rst
> @@ -190,7 +190,7 @@ Toshiba Portege 3020CT		s3_mode (3)
>  Toshiba Satellite 4030CDT	s3_mode (3) (S1 also works OK)
>  Toshiba Satellite 4080XCDT      s3_mode (3) (S1 also works OK)
>  Toshiba Satellite 4090XCDT      ??? [#f1]_
> -Toshiba Satellite P10-554       s3_bios,s3_mode (4)[#f3]_
> +Toshiba Satellite P10-554       s3_bios,s3_mode (4) [#f3]_
>  Toshiba M30                     (2) xor X with nvidia driver using internal AGP

This one is already fixed in docs-next (will go to mainline shortly), so
you can drop this patch.

Thanks,

jon

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Instead of running get_abi.py script, import AbiParser class and
> handle messages directly there using an interactor. This shold save some
> memory, as there's no need to exec python inside the Sphinx python
> extension.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
>  Documentation/sphinx/kernel_abi.py | 26 +++++++++++++++-----------
>  scripts/get_abi.py                 |  2 +-
>  2 files changed, 16 insertions(+), 12 deletions(-)
>
> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> index fc7500fad119..93d537d8cb6c 100644
> --- a/Documentation/sphinx/kernel_abi.py
> +++ b/Documentation/sphinx/kernel_abi.py
> @@ -42,6 +42,11 @@ from docutils.parsers.rst import directives, Directive
>  from sphinx.util.docutils import switch_source_input
>  from sphinx.util import logging
>  
> +srctree = os.path.abspath(os.environ["srctree"])
> +sys.path.insert(0, os.path.join(srctree, "scripts"))
> +
> +from get_abi import AbiParser

I have to admit that I don't like this bit of messing around with the
path.  And importing things out of scripts/ seems ... inelegant.

I take it you still want to be able to run get_abi.py standalone even
after it's directly integrated into Sphinx?  In this case, it might be
nicer to have the common library functionality in its own module that
can be imported into both sphinx and the standalone command.  That still
leaves open the question of where that module lives
(Documentation/sphinx perhaps?) and how the Python path gets set up
correctly...

jon

Re: [RFC v2 00/38] Improve ABI documentation generation

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hi Jon/Greg,
>
> That's the second version of my RFC patches meant to modenize the ABI
> parser that I wrote in Perl.

I have a couple of minor comments on the individual patches, but overall
I do like this direction.

It would be nice, though, if the code were a bit more extensively
commented.  Parts of it get into the "twistly maze of regexes" mode that
can be awfully hard to follow.

> On this series we have:
>
> patches 1 to 11: several bug fixes addressing issues at ABI symbols;

1-3 aren't needed - it seems you already upstreamed #2?

For the rest, is there any reason to not apply them right away?  They
just seem like worthwhile fixes.

> patch 12: a fix for scripts/documentation-file-ref-check
> patches 13-15: create new script with rest and search logic and 
>   minimally integrate with kernel_abi Sphinx extension(phase 1);
> patches 16-19: implement phase 2: class integration (phase 2);
> patch 20: fix a bug at kernel_abi: the way it splits lines is buggy;
> patches  21-24: rewrite kernel_abi logic to make it simpler and more
>   robust;
> patches 25-27: add cross-reference support at automarkup;
> patches 28-36: several ABI cleanups to cope with the improvements;
> patch 37: implement undefined command;
> patch 38: get rid of the old Perl script.
>
> To make it easier to review/apply, I may end breaking the next version
> on a couple of different patchsets. Still it would be nice to have more
> people testing it and providing some feedback.

I've looked over everything, though with limited depth.  My testing
hasn't turned up any problems.  I've only tested with current Sphinx,
have you tried this with the more ancient versions we support?

[It's probably time to raise our minimum version again, especially now
that current Sphinx has better performance.]

I don't see a whole lot of reasons not to apply this set shortly after
the merge window; anybody disagree?

Thanks,

jon

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Mauro Carvalho Chehab]

Em Tue, 28 Jan 2025 15:37:25 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Instead of running get_abi.py script, import AbiParser class and
> > handle messages directly there using an interactor. This shold save some
> > memory, as there's no need to exec python inside the Sphinx python
> > extension.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> >  Documentation/sphinx/kernel_abi.py | 26 +++++++++++++++-----------
> >  scripts/get_abi.py                 |  2 +-
> >  2 files changed, 16 insertions(+), 12 deletions(-)
> >
> > diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> > index fc7500fad119..93d537d8cb6c 100644
> > --- a/Documentation/sphinx/kernel_abi.py
> > +++ b/Documentation/sphinx/kernel_abi.py
> > @@ -42,6 +42,11 @@ from docutils.parsers.rst import directives, Directive
> >  from sphinx.util.docutils import switch_source_input
> >  from sphinx.util import logging
> >  
> > +srctree = os.path.abspath(os.environ["srctree"])
> > +sys.path.insert(0, os.path.join(srctree, "scripts"))
> > +
> > +from get_abi import AbiParser  
> 
> I have to admit that I don't like this bit of messing around with the
> path.  And importing things out of scripts/ seems ... inelegant.
> 
> I take it you still want to be able to run get_abi.py standalone even
> after it's directly integrated into Sphinx?  

Yes, because calling it via command line provides:

1. a way to test the parser and check the results;
2. a search utility;
3. the undefined symbol verification.

Btw, if you look at the other Sphinx modules, they do exactly the same:
they execute code from scripts/. The only difference here is that,
instead of loading the a perl/python/shell engine and running the entire
script from there, it is importing just a class.

> In this case, it might be
> nicer to have the common library functionality in its own module that
> can be imported into both sphinx and the standalone command. 

This would be possible too: place the classes on a common lib dir and
then import it from both command line and Sphinx extensions.

If we're willing to do that, then perhaps we can have separate files
for each different class, as this could make it easier to maintain.

> That still
> leaves open the question of where that module lives
> (Documentation/sphinx perhaps?) and how the Python path gets set up
> correctly...

I guess the command line at scripts/ could use something like this to
get the library location (untested) and add to the import search PATH:

	import os

	python_lib_dir="some/location"

	scriptdir = os.path.dirname(os.path.realpath(__file__))

	sys.path.insert(0, os.path.join(srctree, f"../{python_lib_dir}"))

	from abi_parser import abiParser

Now, I'm not sure if the best location for python libraries would
be at Documentation/sphinx, as we may end needing other python
libraries with time and not all would be used by Sphinx.

In short: I would be more inclined to place them on 
a new lib directory (tools/lib? tools/py_lib? scripts/lib?).

See, with the content of this series, if we split files per each class,
it would mean 3 files:

  1. abi_parser.py, containing AbiParser class (plus ABI_DIR const);
     (this is the only class used by Documentation/sphinx extensions)
  2. abi_regex.py, containing AbiRegex class;
  3. abi_symbols.py, containing SystemSymbols class.

Now, if we're going on this direction, it may also make sense to split
the command line classes/functions into 4 (or 5 files) for argparse
argument definition and command run code. If we do that, it means that
other files will be stored somewhere:

  4. abi_cmd_rest.py: AbiRest and AbiValidate classes for the rest 
     and validate arguments (I would likely place both at the same file,
     as the code is similar - but it could also be split on two separate
     files);
  5. abi_cmd_search.py: AbiSearch - for the search arguments;
  6. abi_cmd_undefined.py: AbiUndefined - for the undocumented symbol check 
     arguments;

Finally, there is the one under scripts/:

  7. get_abi.py: with the main function

For (1), Documentation/sphinx could make sense, but (2) to (6) are
used only by the command line tool. Placing them at Documentation/ 
seems weird. Well, nothing prevents having them at scripts/, IMHO, things
would become more organized if we place the Python files with 0644
flags elsewhere.

Thanks,
Mauro

Re: [RFC v2 00/38] Improve ABI documentation generation

[Mauro Carvalho Chehab]

Em Tue, 28 Jan 2025 15:42:00 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Hi Jon/Greg,
> >
> > That's the second version of my RFC patches meant to modenize the ABI
> > parser that I wrote in Perl.  
> 
> I have a couple of minor comments on the individual patches, but overall
> I do like this direction.
> 
> It would be nice, though, if the code were a bit more extensively
> commented.  Parts of it get into the "twistly maze of regexes" mode that
> can be awfully hard to follow.

The regex code is indeed complex, but documenting it is not an easy task.
Btw, they are (about) the same that the Perl script does. imported also
the documentation for there. I did some extra cleanups/optimizations there,
though, after checking the results of some expressions.

The big issue is that we don't have an uniform way of defining What: 
expressions. So, each subsystem (and/or author) document it in different
ways.

There are even some ABI symbols with:

	$(readlink)/sys/...

(I intend to send a patch for those later on)

and:

	What: /sys/something/...

	What: .../something_else

(I guess ".../" means "/sys/something/...", but I can't be sure, as this
is on one driver for a hardware I don't have - so, if I send a patch,
I may end breaking it)

If you want to understand how the whole set of regexes work, you can
run:

	$ ./scripts/get_abi.py -d 16 undefined --dry-run >/dev/null
...
	[DEBUG] /sys/kernel/mm/damon/admin/kdamonds/\w+/contexts/\w+/schemes/\w+/quotas/goals/\w+/current_value <== /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/current_value
	[DEBUG] /sys/kernel/mm/damon/admin/kdamonds/\w+/contexts/\w+/schemes/\w+/quotas/goals/\w+/target_metric <== /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/target_metric
	[DEBUG] /sys/kernel/mm/damon/admin/kdamonds/\w+/contexts/\w+/schemes/\w+/quotas/goals/\w+/target_value <== /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/<G>/target_value
	[DEBUG] /sys/kernel/mm/damon/admin/kdamonds/\w+/contexts/\w+/schemes/\w+/quotas/goals/nr_goals     <== /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/goals/nr_goals
	[DEBUG] /sys/kernel/mm/damon/admin/kdamonds/\w+/contexts/\w+/schemes/\w+/quotas/ms                 <== /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/ms
	[DEBUG] /sys/kernel/mm/damon/admin/kdamonds/\w+/contexts/\w+/schemes/\w+/quotas/reset_interval_ms  <== /sys/kernel/mm/
...

This will place at stderr all regular expressions that are currently
parsed (they're currently used only for /sys symbols).

Yet, instead of spending too much time documenting them, IMO we shold
do the do the reverse: use the AbiRegex class to convert "What:" into
a new tag (like "Regex:") and use it as much as possible (we'll still
need "What:" for some things that aren't devnodes), as, with regular
expressions, symbols can be clearly documented. As on python match groups
can be named with:

	(?P<name>...)

this could be used to better describe some arguments, e.g. (picking an
easy case):

	What: /sys/module/<MODULENAME>/srcversion

could be described, instead, as:

	Regex: /sys/module/(?P<MODULENAME>[\w\-]+)/srcversion

The Kernel_abi extension (actually AbiParser class) can either display it
as-is (my personal preference), or even replace:
	(?P<MODULENAME>[\w\-]+)
with:
	MODULENAME

and still output this at html/pdf output as before, e. g.:

	What: /sys/module/<MODULENAME>/srcversion

Yet, doing it on a consistent way.

This is easier said than done, as if we do some automatic conversion,
subsystem reviewers/maintainers will need to double-check if the
converted expressions make sense.


> > On this series we have:
> >
> > patches 1 to 11: several bug fixes addressing issues at ABI symbols;  
> 
> 1-3 aren't needed - it seems you already upstreamed #2?
> 
> For the rest, is there any reason to not apply them right away?  They
> just seem like worthwhile fixes.
> 
> > patch 12: a fix for scripts/documentation-file-ref-check
> > patches 13-15: create new script with rest and search logic and 
> >   minimally integrate with kernel_abi Sphinx extension(phase 1);
> > patches 16-19: implement phase 2: class integration (phase 2);
> > patch 20: fix a bug at kernel_abi: the way it splits lines is buggy;
> > patches  21-24: rewrite kernel_abi logic to make it simpler and more
> >   robust;
> > patches 25-27: add cross-reference support at automarkup;
> > patches 28-36: several ABI cleanups to cope with the improvements;
> > patch 37: implement undefined command;
> > patch 38: get rid of the old Perl script.
> >
> > To make it easier to review/apply, I may end breaking the next version
> > on a couple of different patchsets. Still it would be nice to have more
> > people testing it and providing some feedback.  
> 
> I've looked over everything, though with limited depth. 

> My testing hasn't turned up any problems.  

Great!

> I've only tested with current Sphinx,
> have you tried this with the more ancient versions we support?

Not yet, but I double-checked at Sphinx documentation to be sure that
I won't be using any newer methods: I just kept using the same Sphinx
API as used by other extensions at the Kernel.

For instance this loop:

    def do_parse(self, content, node):
        with switch_source_input(self.state, content):
            self.state.nested_parse(content, 0, node, match_titles=1)

was changed on Sphinx 7.4[1], and even nested_parse(match_titles=1) is
not the recommended code for versions < 7.4, as there is this 
replacement function:

	nested_parse_with_titles()

Yet, as they're working fine at least up to version 8.1.3, we can
keep using the old way.

In any case, I'll do a test before sending the final version to see if
it works fine with our minimal version.

[1] See: https://www.sphinx-doc.org/en/master/extdev/markupapi.html

- 

On a separate discussion, I noticed one potential compatibility issue
we may have with future Python versions, due to some ascii texts
formatted as unicode. I'll send later a patch fixing them.

Additionally, automarkup has backward-compatible code with Python 2.7.
 Can I send patches dropping 2.7 support from Sphinx extensions?

> [It's probably time to raise our minimum version again, especially now
> that current Sphinx has better performance.]

Agreed. 

IMO, we should also increase Python's minimal version.

> I don't see a whole lot of reasons not to apply this set shortly after
> the merge window; anybody disagree?

Thanks,
Mauro

Re: [RFC v2 17/38] scripts/get_abi.py: use an interactor for ReST output

[Akira Yokosawa]

Hi,

Mauro Carvalho Chehab wrote:

> Instead of printing all results line per line, use an interactor
> to return each variable as a separate message.
> 
> This won't change much when using it via command line, but it
> will help Sphinx integration by providing an interactor that
> could be used there to handle ABI symbol by symbol.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
>  scripts/get_abi.py | 52 ++++++++++++++++++++++++++--------------------
>  1 file changed, 29 insertions(+), 23 deletions(-)
> 
> diff --git a/scripts/get_abi.py b/scripts/get_abi.py
> index 73613fb29c1c..2aec1f9dc5aa 100755
> --- a/scripts/get_abi.py
> +++ b/scripts/get_abi.py
[...]
> @@ -470,9 +472,9 @@ class AbiParser:
>  
>                  if cur_part and cur_part != part:
>                      part = cur_part
> -                    print(f"{part}\n{"-" * len(part)}\n")
> +                    msg += f"{part}\n{"-" * len(part)}\n\n"
>  
> -                print(f".. _{key}:\n")
> +                msg += f".. _{key}:\n\n"
>  
>                  max_len = 0
>                  for i in range(0, len(names)):           # pylint: disable=C0200
[...]

Testing under Ubuntu 22.04, where distro python3 is 3.10.12 and
distro Sphinx is 4.3.2, I get this exception by running "make htmldocs":

Exception occurred:
  File ".../linux/Documentation/sphinx/kernel_abi.py", line 48, in <module>
    from get_abi import AbiParser
  File ".../linux/scripts/get_abi.py", line 525
    msg += f"{part}\n{"-" * len(part)}\n\n"
                       ^
SyntaxError: f-string: expecting '}'

, which is introduced in the above hunk, I guess.

You can install Sphinx 8.1.3 on top of python3 3.10.12 using venv.
I get the same exception there.

Your change works fine against Ubuntu 24.04, whose distro python3 is 3.12.3.

I have not tested against python3 3.11.x.

It would be better to keep compatible with >= python 3.10.x if at all
possible.

        Thanks, Akira

Re: [RFC v2 17/38] scripts/get_abi.py: use an interactor for ReST output

[Mauro Carvalho Chehab]

Em Wed, 29 Jan 2025 20:04:42 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> Hi,
> 
> Mauro Carvalho Chehab wrote:
> 
> > Instead of printing all results line per line, use an interactor
> > to return each variable as a separate message.
> > 
> > This won't change much when using it via command line, but it
> > will help Sphinx integration by providing an interactor that
> > could be used there to handle ABI symbol by symbol.
> > 
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> >  scripts/get_abi.py | 52 ++++++++++++++++++++++++++--------------------
> >  1 file changed, 29 insertions(+), 23 deletions(-)
> > 
> > diff --git a/scripts/get_abi.py b/scripts/get_abi.py
> > index 73613fb29c1c..2aec1f9dc5aa 100755
> > --- a/scripts/get_abi.py
> > +++ b/scripts/get_abi.py  
> [...]
> > @@ -470,9 +472,9 @@ class AbiParser:
> >  
> >                  if cur_part and cur_part != part:
> >                      part = cur_part
> > -                    print(f"{part}\n{"-" * len(part)}\n")
> > +                    msg += f"{part}\n{"-" * len(part)}\n\n"
> >  
> > -                print(f".. _{key}:\n")
> > +                msg += f".. _{key}:\n\n"
> >  
> >                  max_len = 0
> >                  for i in range(0, len(names)):           # pylint: disable=C0200  
> [...]
> 
> Testing under Ubuntu 22.04, where distro python3 is 3.10.12 and
> distro Sphinx is 4.3.2, I get this exception by running "make htmldocs":
> 
> Exception occurred:
>   File ".../linux/Documentation/sphinx/kernel_abi.py", line 48, in <module>
>     from get_abi import AbiParser
>   File ".../linux/scripts/get_abi.py", line 525
>     msg += f"{part}\n{"-" * len(part)}\n\n"
>                        ^
> SyntaxError: f-string: expecting '}'
> 
> , which is introduced in the above hunk, I guess.
> 
> You can install Sphinx 8.1.3 on top of python3 3.10.12 using venv.
> I get the same exception there.
> 
> Your change works fine against Ubuntu 24.04, whose distro python3 is 3.12.3.
> 
> I have not tested against python3 3.11.x.
> 
> It would be better to keep compatible with >= python 3.10.x if at all
> possible.

Thanks for checking it!

I was aiming to make it compatible with 3.6. Yet, it seems that f-string
support is very limited on early versions. The enclosed diff will make
it backward-compatible with Python 3.6.

Thanks,
Mauro

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 543bed397c8c..e6e94f721fff 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -522,7 +522,7 @@ class AbiParser:
 
                 if cur_part and cur_part != part:
                     part = cur_part
-                    msg += f"{part}\n{"-" * len(part)}\n\n"
+                    msg += part + "\n"+ "-" * len(part) +"\n\n"
 
                 msg += f".. _{key}:\n\n"
 
@@ -546,7 +546,7 @@ class AbiParser:
                     msg += f"Defined on file :ref:`{base} <{ref[1]}>`\n\n"
 
             if wtype == "File":
-                msg += f"{names[0]}\n{"-" * len(names[0])}\n\n"
+                msg += names[0] +"\n" + "-" * len(names[0]) +"\n\n"
 
             desc = v.get("description")
             if not desc and wtype != "File":
@@ -570,7 +570,8 @@ class AbiParser:
 
             users = v.get("users")
             if users and users.strip(" \t\n"):
-                msg += f"Users:\n\t{users.strip("\n").replace('\n', '\n\t')}\n\n"
+                users = users.strip("\n").replace('\n', '\n\t')
+                msg += f"Users:\n\t{users}\n\n"
 
             ln = v.get("line_no", 1)
 
@@ -596,7 +597,9 @@ class AbiParser:
                 elif len(lines) == 1:
                     f.append(f"{fname}:{lines[0]}")
                 else:
-                    f.append(f"{fname} lines {", ".join(str(x) for x in lines)}")
+                    m = fname + "lines "
+                    m += ", ".join(str(x) for x in lines)
+                    f.append(m)
 
             self.log.warning("%s is defined %d times: %s", what, len(f), "; ".join(f))
 
@@ -644,10 +647,11 @@ class AbiParser:
                     if users:
                         print(f"Users:\t\t\t{users}")
 
-                    print(f"Defined on file{'s'[:len(files) ^ 1]}:\t{", ".join(files)}")
+                    print("Defined on file(s):\t" + ", ".join(files))
 
                     if desc:
-                        print(f"\n{desc.strip("\n")}\n")
+                        desc = desc.strip("\n")
+                        print(f"\n{desc}\n")
 
         if not found_keys:
             print(f"Regular expression /{expr}/ not found.")

Re: [RFC v2 00/38] Improve ABI documentation generation

[Mauro Carvalho Chehab]

Em Wed, 29 Jan 2025 02:45:18 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> > I've only tested with current Sphinx,
> > have you tried this with the more ancient versions we support?  
> 
> Not yet, but I double-checked at Sphinx documentation to be sure that
> I won't be using any newer methods: I just kept using the same Sphinx
> API as used by other extensions at the Kernel.

Just checked it with Python 3.6 and Sphinx 3.4 on Fedora 41 with:

	sudo dnf install python3.6.x86_64
	python3.6 -m venv Sphinx-3.4
	pip install alabaster Sphinx==3.4.3 pyyaml

There were some issues related to problems with early f-string
support, as reported by Akira.

After applying the enclosed patch, it is now working fine. The only
drawback is here:

	- print(f"Defined on file{'s'[:len(files) ^ 1]}:\t{", ".join(files)}")
	+ print("Defined on file(s):\t" + ", ".join(files))

As I removed the file/files auto-plural logic depending on the files
array length. Not a big deal.

I'll double-check if there's no other diff between old/new version
and add the enclosed patch in the end.

Thanks,
Mauro

[PATCH] scripts/get_abi.py: make it backward-compatible with Python 3.6

Despite being introduced on Python 3.6, the original implementation
was too limited: it doesn't accept anything but the argument.

Even on python 3.10.12, support was still limited, as more complex
operations cause SyntaxError:

	Exception occurred:
	  File ".../linux/Documentation/sphinx/kernel_abi.py", line 48, in <module>
	    from get_abi import AbiParser
	  File ".../linux/scripts/get_abi.py", line 525
	    msg += f"{part}\n{"-" * len(part)}\n\n"
                       ^
	SyntaxError: f-string: expecting '}'

Replace f-strings by normal string concatenation when it doesn't
work on Python 3.6.

Reported-by: Akira Yokosawa <akiyks@gmail.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

diff --git a/scripts/get_abi.py b/scripts/get_abi.py
index 543bed397c8c..e6e94f721fff 100755
--- a/scripts/get_abi.py
+++ b/scripts/get_abi.py
@@ -522,7 +522,7 @@ class AbiParser:
 
                 if cur_part and cur_part != part:
                     part = cur_part
-                    msg += f"{part}\n{"-" * len(part)}\n\n"
+                    msg += part + "\n"+ "-" * len(part) +"\n\n"
 
                 msg += f".. _{key}:\n\n"
 
@@ -546,7 +546,7 @@ class AbiParser:
                     msg += f"Defined on file :ref:`{base} <{ref[1]}>`\n\n"
 
             if wtype == "File":
-                msg += f"{names[0]}\n{"-" * len(names[0])}\n\n"
+                msg += names[0] +"\n" + "-" * len(names[0]) +"\n\n"
 
             desc = v.get("description")
             if not desc and wtype != "File":
@@ -570,7 +570,8 @@ class AbiParser:
 
             users = v.get("users")
             if users and users.strip(" \t\n"):
-                msg += f"Users:\n\t{users.strip("\n").replace('\n', '\n\t')}\n\n"
+                users = users.strip("\n").replace('\n', '\n\t')
+                msg += f"Users:\n\t{users}\n\n"
 
             ln = v.get("line_no", 1)
 
@@ -596,7 +597,9 @@ class AbiParser:
                 elif len(lines) == 1:
                     f.append(f"{fname}:{lines[0]}")
                 else:
-                    f.append(f"{fname} lines {", ".join(str(x) for x in lines)}")
+                    m = fname + "lines "
+                    m += ", ".join(str(x) for x in lines)
+                    f.append(m)
 
             self.log.warning("%s is defined %d times: %s", what, len(f), "; ".join(f))
 
@@ -644,10 +647,11 @@ class AbiParser:
                     if users:
                         print(f"Users:\t\t\t{users}")
 
-                    print(f"Defined on file{'s'[:len(files) ^ 1]}:\t{", ".join(files)}")
+                    print("Defined on file(s):\t" + ", ".join(files))
 
                     if desc:
-                        print(f"\n{desc.strip("\n")}\n")
+                        desc = desc.strip("\n")
+                        print(f"\n{desc}\n")
 
         if not found_keys:
             print(f"Regular expression /{expr}/ not found.")

Re: [RFC v2 00/38] Improve ABI documentation generation

[Mauro Carvalho Chehab]

Em Tue, 28 Jan 2025 15:42:00 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> [It's probably time to raise our minimum version again, especially now
> that current Sphinx has better performance.]

Last change was about one year ago, so it sounds fair to also
change Sphinx minimal version with about one year gap. What we have
currently is:

	2.4.4: minimal version
	3.4.3: suggested minimal version.

Looking at the relevant release dates, we have:

	Release 2.4.0 (released Feb 09, 2020)
	Release 2.4.4 (released Mar 05, 2020)
	Release 3.4.0 (released Dec 20, 2020)
	Release 3.4.3 (released Jan 08, 2021)

So, the ~one year gap is what it takes to raise the bar from 2.4.4 to
3.4.3.

In terms of Python, we're currently at 3.5:

	Python	Release date 
	3.5	2015-09-13
	3.6	2016-12-23
	3.7 	2018-06-27
	3.8 	2019-10-14
	3.9 	2020-10-05
	3.10	2021-10-04

	(according with https://en.wikipedia.org/w/index.php?title=History_of_Python) 

Python 3.6 is the first one with f-string support, with is something
that most Python programmers use those days. So, IMO, that would
be the absolute minimal version we should pick.

Yet, IMHO, we should aim to be backard-compatible with the tools 
available up to a certain date (Jan, 2021) - e. g. we'll aim to
support at least a 4 years old toolset for documentation build.

So, I'm proposing to change the minimal requirements to:
	- Sphinx 3.4.3;
	- Python 3.9

By setting Sphinx minimal version to 3.4.3, we can get rid of all
Sphinx backward-compatible code.

I have already patches with such changes for it on the top of this RFC. 
Will send it shortly as RFC, aiming to send the final version after -rc1.

Thanks,
Mauro

Re: [RFC v2 00/38] Improve ABI documentation generation

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> So, I'm proposing to change the minimal requirements to:
> 	- Sphinx 3.4.3;
> 	- Python 3.9
>
> By setting Sphinx minimal version to 3.4.3, we can get rid of all
> Sphinx backward-compatible code.

That's certainly a nice thought.

With regard to Python ... are all reasonable distributions at 3.9 at
least?  CentOS 9 seems to be there, and Debian beyond it.  So probably
that is a reasonable floor to set?

jon

Re: [RFC v2 00/38] Improve ABI documentation generation

[Mauro Carvalho Chehab]

Em Wed, 29 Jan 2025 08:58:13 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > So, I'm proposing to change the minimal requirements to:
> > 	- Sphinx 3.4.3;
> > 	- Python 3.9
> >
> > By setting Sphinx minimal version to 3.4.3, we can get rid of all
> > Sphinx backward-compatible code.  
> 
> That's certainly a nice thought.
> 
> With regard to Python ... are all reasonable distributions at 3.9 at
> least?  CentOS 9 seems to be there, and Debian beyond it.  So probably
> that is a reasonable floor to set?

I didn't check, but those are the current minimal versions above 3.5 for
what we have at the Kernel tree[1]:

            !2, 3.10     tools/net/sunrpc/xdrgen/generators/__init__.py
            !2, 3.10     tools/net/sunrpc/xdrgen/generators/program.py
            !2, 3.10     tools/net/sunrpc/xdrgen/subcmds/source.py
            !2, 3.10     tools/net/sunrpc/xdrgen/xdr_ast.py
            !2, 3.10     tools/power/cpupower/bindings/python/test_raw_pylibcpupower.py
            !2, 3.9      tools/testing/selftests/net/rds/test.py
            !2, 3.9      tools/net/ynl/ethtool.py
            !2, 3.9      tools/net/ynl/cli.py
            !2, 3.9      scripts/checktransupdate.py
            !2, 3.8      tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py
            !2, 3.8      tools/testing/selftests/hid/tests/base.py
            !2, 3.7      tools/testing/selftests/turbostat/smi_aperf_mperf.py
            !2, 3.7      tools/testing/selftests/turbostat/defcolumns.py
            !2, 3.7      tools/testing/selftests/turbostat/added_perf_counters.py
            !2, 3.7      tools/testing/selftests/hid/tests/conftest.py
            !2, 3.7      tools/testing/kunit/qemu_config.py
            !2, 3.7      tools/testing/kunit/kunit_tool_test.py
            !2, 3.7      tools/testing/kunit/kunit.py
            !2, 3.7      tools/testing/kunit/kunit_parser.py
            !2, 3.7      tools/testing/kunit/kunit_kernel.py
            !2, 3.7      tools/testing/kunit/kunit_json.py
            !2, 3.7      tools/testing/kunit/kunit_config.py
            !2, 3.7      tools/perf/scripts/python/gecko.py
            !2, 3.7      scripts/rust_is_available_test.py
            !2, 3.7      scripts/bpf_doc.py
            !2, 3.6      tools/writeback/wb_monitor.py
            !2, 3.6      tools/workqueue/wq_monitor.py
            !2, 3.6      tools/workqueue/wq_dump.py
            !2, 3.6      tools/usb/p9_fwd.py
            !2, 3.6      tools/tracing/rtla/sample/timerlat_load.py
            !2, 3.6      tools/testing/selftests/net/openvswitch/ovs-dpctl.py
            !2, 3.6      tools/testing/selftests/net/nl_netdev.py
            !2, 3.6      tools/testing/selftests/net/lib/py/ynl.py
            !2, 3.6      tools/testing/selftests/net/lib/py/utils.py
            !2, 3.6      tools/testing/selftests/net/lib/py/nsim.py
            !2, 3.6      tools/testing/selftests/net/lib/py/netns.py
            !2, 3.6      tools/testing/selftests/net/lib/py/ksft.py
            !2, 3.6      tools/testing/selftests/kselftest/ksft.py
            !2, 3.6      tools/testing/selftests/hid/tests/test_tablet.py
            !2, 3.6      tools/testing/selftests/hid/tests/test_sony.py
            !2, 3.6      tools/testing/selftests/hid/tests/test_multitouch.py
            !2, 3.6      tools/testing/selftests/hid/tests/test_mouse.py
            !2, 3.6      tools/testing/selftests/hid/tests/base_gamepad.py
            !2, 3.6      tools/testing/selftests/hid/tests/base_device.py
            !2, 3.6      tools/testing/selftests/drivers/net/stats.py
            !2, 3.6      tools/testing/selftests/drivers/net/shaper.py
            !2, 3.6      tools/testing/selftests/drivers/net/queues.py
            !2, 3.6      tools/testing/selftests/drivers/net/ping.py
            !2, 3.6      tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
            !2, 3.6      tools/testing/selftests/drivers/net/lib/py/load.py
            !2, 3.6      tools/testing/selftests/drivers/net/lib/py/__init__.py
            !2, 3.6      tools/testing/selftests/drivers/net/lib/py/env.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/rss_ctx.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/pp_alloc_fail.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/nic_performance.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/nic_link_layer.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/lib/py/linkconfig.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/lib/py/__init__.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/devmem.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/devlink_port_split.py
            !2, 3.6      tools/testing/selftests/drivers/net/hw/csum.py
            !2, 3.6      tools/testing/selftests/devices/probe/test_discoverable_devices.py
            !2, 3.6      tools/testing/selftests/bpf/test_bpftool_synctypes.py
            !2, 3.6      tools/testing/selftests/bpf/generate_udp_fragments.py
            !2, 3.6      tools/testing/kunit/run_checks.py
            !2, 3.6      tools/testing/kunit/kunit_printer.py
            !2, 3.6      tools/sched_ext/scx_show_state.py
            !2, 3.6      tools/perf/tests/shell/lib/perf_metric_validation.py
            !2, 3.6      tools/perf/tests/shell/lib/perf_json_output_lint.py
            !2, 3.6      tools/perf/scripts/python/parallel-perf.py
            !2, 3.6      tools/perf/scripts/python/flamegraph.py
            !2, 3.6      tools/perf/scripts/python/arm-cs-trace-disasm.py
            !2, 3.6      tools/perf/pmu-events/models.py
            !2, 3.6      tools/perf/pmu-events/metric_test.py
            !2, 3.6      tools/perf/pmu-events/metric.py
            !2, 3.6      tools/perf/pmu-events/jevents.py
            !2, 3.6      tools/net/ynl/ynl-gen-rst.py
            !2, 3.6      tools/net/ynl/ynl-gen-c.py
            !2, 3.6      tools/net/ynl/lib/ynl.py
            !2, 3.6      tools/net/ynl/lib/nlspec.py
            !2, 3.6      tools/crypto/tcrypt/tcrypt_speed_compare.py
            !2, 3.6      tools/cgroup/iocost_monitor.py
            !2, 3.6      tools/cgroup/iocost_coef_gen.py
            !2, 3.6      scripts/make_fit.py
            !2, 3.6      scripts/macro_checker.py
            !2, 3.6      scripts/get_abi.py
            !2, 3.6      scripts/generate_rust_analyzer.py
            !2, 3.6      scripts/gdb/linux/timerlist.py
            !2, 3.6      scripts/gdb/linux/pgtable.py
            !2, 3.6      scripts/clang-tools/run-clang-tools.py
            !2, 3.6      Documentation/sphinx/automarkup.py

[1] Checked with:
	vermin -v $(git ls-files *.py)

    Please notice that vermin is not perfect: my script passed as version 3.6
    because the f-string check there didn't verify f-string improvements over
    time. Still, it is a quick way to check that our current minimal version
    is not aligned with reality.
 
Btw, vermin explains what is requiring more at the scripts. For instance:

	$ vermin -vv scripts/checktransupdate.py
	...
	!2, 3.9      /new_devel/v4l/docs/scripts/checktransupdate.py
	  'argparse' module requires 2.7, 3.2
	  'argparse.BooleanOptionalAction' member requires !2, 3.9
	  'datetime' module requires 2.3, 3.0
	  'datetime.datetime.strptime' member requires 2.5, 3.0
	  'logging' module requires 2.3, 3.0
	  'logging.StreamHandler' member requires 2.6, 3.0
	  'os.path.relpath' member requires 2.6, 3.0
	  f-strings require !2, 3.6

Thanks,
Mauro

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Mauro Carvalho Chehab]

Hi Jon,

Em Wed, 29 Jan 2025 01:43:24 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Em Tue, 28 Jan 2025 15:37:25 -0700
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >   
> > > Instead of running get_abi.py script, import AbiParser class and
> > > handle messages directly there using an interactor. This shold save some
> > > memory, as there's no need to exec python inside the Sphinx python
> > > extension.
> > >
> > > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > > ---
> > >  Documentation/sphinx/kernel_abi.py | 26 +++++++++++++++-----------
> > >  scripts/get_abi.py                 |  2 +-
> > >  2 files changed, 16 insertions(+), 12 deletions(-)
> > >
> > > diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> > > index fc7500fad119..93d537d8cb6c 100644
> > > --- a/Documentation/sphinx/kernel_abi.py
> > > +++ b/Documentation/sphinx/kernel_abi.py
> > > @@ -42,6 +42,11 @@ from docutils.parsers.rst import directives, Directive
> > >  from sphinx.util.docutils import switch_source_input
> > >  from sphinx.util import logging
> > >  
> > > +srctree = os.path.abspath(os.environ["srctree"])
> > > +sys.path.insert(0, os.path.join(srctree, "scripts"))
> > > +
> > > +from get_abi import AbiParser    
> > 
> > I have to admit that I don't like this bit of messing around with the
> > path.  And importing things out of scripts/ seems ... inelegant.
> > 
> > I take it you still want to be able to run get_abi.py standalone even
> > after it's directly integrated into Sphinx?    
> 
> Yes, because calling it via command line provides:
> 
> 1. a way to test the parser and check the results;
> 2. a search utility;
> 3. the undefined symbol verification.
> 
> Btw, if you look at the other Sphinx modules, they do exactly the same:
> they execute code from scripts/. The only difference here is that,
> instead of loading the a perl/python/shell engine and running the entire
> script from there, it is importing just a class.
> 
> > In this case, it might be
> > nicer to have the common library functionality in its own module that
> > can be imported into both sphinx and the standalone command.   
> 
> This would be possible too: place the classes on a common lib dir and
> then import it from both command line and Sphinx extensions.
> 
> If we're willing to do that, then perhaps we can have separate files
> for each different class, as this could make it easier to maintain.
> 
> > That still
> > leaves open the question of where that module lives
> > (Documentation/sphinx perhaps?) and how the Python path gets set up
> > correctly...  
> 
> I guess the command line at scripts/ could use something like this to
> get the library location (untested) and add to the import search PATH:
> 
> 	import os
> 
> 	python_lib_dir="some/location"
> 
> 	scriptdir = os.path.dirname(os.path.realpath(__file__))
> 
> 	sys.path.insert(0, os.path.join(srctree, f"../{python_lib_dir}"))
> 
> 	from abi_parser import abiParser
> 
> Now, I'm not sure if the best location for python libraries would
> be at Documentation/sphinx, as we may end needing other python
> libraries with time and not all would be used by Sphinx.
> 
> In short: I would be more inclined to place them on 
> a new lib directory (tools/lib? tools/py_lib? scripts/lib?).
> 
> See, with the content of this series, if we split files per each class,
> it would mean 3 files:
> 
>   1. abi_parser.py, containing AbiParser class (plus ABI_DIR const);
>      (this is the only class used by Documentation/sphinx extensions)
>   2. abi_regex.py, containing AbiRegex class;
>   3. abi_symbols.py, containing SystemSymbols class.
> 
> Now, if we're going on this direction, it may also make sense to split
> the command line classes/functions into 4 (or 5 files) for argparse
> argument definition and command run code. If we do that, it means that
> other files will be stored somewhere:
> 
>   4. abi_cmd_rest.py: AbiRest and AbiValidate classes for the rest 
>      and validate arguments (I would likely place both at the same file,
>      as the code is similar - but it could also be split on two separate
>      files);
>   5. abi_cmd_search.py: AbiSearch - for the search arguments;
>   6. abi_cmd_undefined.py: AbiUndefined - for the undocumented symbol check 
>      arguments;
> 
> Finally, there is the one under scripts/:
> 
>   7. get_abi.py: with the main function
> 
> For (1), Documentation/sphinx could make sense, but (2) to (6) are
> used only by the command line tool. Placing them at Documentation/ 
> seems weird. Well, nothing prevents having them at scripts/, IMHO, things
> would become more organized if we place the Python files with 0644
> flags elsewhere.

As I'll be preparing such patches for merge along this week, I'd
like to know what do you prefer in terms of directories:

1. Keep it as-is;
2. have a separate library directory for Python modules
   (scripts/lib?);
3. place python modules inside scripts/;
4. place python modules inside Documentation/sphinx (IMO a bad
   idea);
5. something else

-

Btw, I'm considering to also submit later a patchset similar to
this one converting kernel-doc to Python. I already started writing
something like that (written from the scratch, following as much
as possible what we have today on Perl to avoid regressions).

I would probably split the code into separate classes to make the code 
more readable/maintainable (a base class, a class with rest output,
another one with man output, and a few other helper classes).

Thanks,
Mauro

Re: [RFC v2 08/38] ABI: sysfs-bus-coresight-*: fix kernelversion tags

[Suzuki K Poulose]

On 28/01/2025 00:05, Mauro Carvalho Chehab wrote:
> Some kernelversion tags are missing colons. Add them to comply with
> ABI description and produce right results when converted to html/pdf.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

Acked-by: Suzuki K Poulose <suzuki.poulose@arm.com>

> ---
>   .../testing/sysfs-bus-coresight-devices-cti   | 78 +++++++++----------
>   .../testing/sysfs-bus-coresight-devices-tpdm  | 52 ++++++-------
>   2 files changed, 65 insertions(+), 65 deletions(-)
> 
> diff --git a/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti b/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti
> index bf2869c413e7..a97b70f588da 100644
> --- a/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti
> +++ b/Documentation/ABI/testing/sysfs-bus-coresight-devices-cti
> @@ -1,241 +1,241 @@
>   What:		/sys/bus/coresight/devices/<cti-name>/enable
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Enable/Disable the CTI hardware.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/powered
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Indicate if the CTI hardware is powered.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/ctmid
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Display the associated CTM ID
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/nr_trigger_cons
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Number of devices connected to triggers on this CTI
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/name
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Name of connected device <N>
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/in_signals
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Input trigger signals from connected device <N>
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/in_types
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Functional types for the input trigger signals
>   		from connected device <N>
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/out_signals
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Output trigger signals to connected device <N>
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/triggers<N>/out_types
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Functional types for the output trigger signals
>   		to connected device <N>
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/inout_sel
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Select the index for inen and outen registers.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/inen
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Read or write the CTIINEN register selected by inout_sel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/outen
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Read or write the CTIOUTEN register selected by inout_sel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/gate
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Read or write CTIGATE register.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/asicctl
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Read or write ASICCTL register.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/intack
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Write the INTACK register.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/appset
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Set CTIAPPSET register to activate channel. Read back to
>   		determine current value of register.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/appclear
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Write APPCLEAR register to deactivate channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/apppulse
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Write APPPULSE to pulse a channel active for one clock
>   		cycle.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/chinstatus
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Read current status of channel inputs.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/choutstatus
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) read current status of channel outputs.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/triginstatus
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) read current status of input trigger signals
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/regs/trigoutstatus
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) read current status of output trigger signals.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/trigin_attach
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Attach a CTI input trigger to a CTM channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/trigin_detach
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Detach a CTI input trigger from a CTM channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/trigout_attach
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Attach a CTI output trigger to a CTM channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/trigout_detach
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Detach a CTI output trigger from a CTM channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_gate_enable
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Enable CTIGATE for single channel (Write) or list enabled
>   		channels through the gate (R).
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_gate_disable
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Disable CTIGATE for single channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_set
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Activate a single channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_clear
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Deactivate a single channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_pulse
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Pulse a single channel - activate for a single clock cycle.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/trigout_filtered
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) List of output triggers filtered across all connections.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/trig_filter_enable
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Enable or disable trigger output signal filtering.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_inuse
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) show channels with at least one attached trigger signal.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_free
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) show channels with no attached trigger signals.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_sel
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(RW) Write channel number to select a channel to view, read to
>   		see selected channel number.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_in
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Read to see input triggers connected to selected view
>   		channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_out
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Read) Read to see output triggers connected to selected view
>   		channel.
>   
>   What:		/sys/bus/coresight/devices/<cti-name>/channels/chan_xtrigs_reset
>   Date:		March 2020
> -KernelVersion	5.7
> +KernelVersion:	5.7
>   Contact:	Mike Leach or Mathieu Poirier
>   Description:	(Write) Clear all channel / trigger programming.
> diff --git a/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm b/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm
> index bf710ea6e0ef..53cb454b60d0 100644
> --- a/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm
> +++ b/Documentation/ABI/testing/sysfs-bus-coresight-devices-tpdm
> @@ -1,6 +1,6 @@
>   What:		/sys/bus/coresight/devices/<tpdm-name>/integration_test
>   Date:		January 2023
> -KernelVersion	6.2
> +KernelVersion:	6.2
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(Write) Run integration test for tpdm. Integration test
> @@ -14,7 +14,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/reset_dataset
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(Write) Reset the dataset of the tpdm.
> @@ -24,7 +24,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_type
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the trigger type of the DSB for tpdm.
> @@ -35,7 +35,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_ts
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the trigger timestamp of the DSB for tpdm.
> @@ -46,7 +46,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_mode
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the programming mode of the DSB for tpdm.
> @@ -60,7 +60,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/ctrl_idx
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the index number of the edge detection for the DSB
> @@ -69,7 +69,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/ctrl_val
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		Write a data to control the edge detection corresponding to
> @@ -85,7 +85,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/ctrl_mask
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		Write a data to mask the edge detection corresponding to the index
> @@ -97,21 +97,21 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/edcr[0:15]
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		Read a set of the edge control value of the DSB in TPDM.
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_edge/edcmr[0:7]
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		Read a set of the edge control mask of the DSB in TPDM.
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_patt/xpr[0:7]
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the value of the trigger pattern for the DSB
> @@ -119,7 +119,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_trig_patt/xpmr[0:7]
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the mask of the trigger pattern for the DSB
> @@ -127,21 +127,21 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpr[0:7]
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the value of the pattern for the DSB subunit TPDM.
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpmr[0:7]
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the mask of the pattern for the DSB subunit TPDM.
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/enable_ts
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(Write) Set the pattern timestamp of DSB tpdm. Read
> @@ -153,7 +153,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/set_type
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(Write) Set the pattern type of DSB tpdm. Read
> @@ -165,7 +165,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_msr/msr[0:31]
>   Date:		March 2023
> -KernelVersion	6.7
> +KernelVersion:	6.7
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the MSR(mux select register) for the DSB subunit
> @@ -173,7 +173,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_mode
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:	(Write) Set the data collection mode of CMB tpdm. Continuous
>   		change creates CMB data set elements on every CMBCLK edge.
> @@ -187,7 +187,7 @@ Description:	(Write) Set the data collection mode of CMB tpdm. Continuous
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_trig_patt/xpr[0:1]
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the value of the trigger pattern for the CMB
> @@ -195,7 +195,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_trig_patt/xpmr[0:1]
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the mask of the trigger pattern for the CMB
> @@ -203,21 +203,21 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpr[0:1]
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the value of the pattern for the CMB subunit TPDM.
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/dsb_patt/tpmr[0:1]
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the mask of the pattern for the CMB subunit TPDM.
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_patt/enable_ts
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(Write) Set the pattern timestamp of CMB tpdm. Read
> @@ -229,7 +229,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_trig_ts
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the trigger timestamp of the CMB for tpdm.
> @@ -240,7 +240,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_ts_all
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Read or write the status of timestamp upon all interface.
> @@ -252,7 +252,7 @@ Description:
>   
>   What:		/sys/bus/coresight/devices/<tpdm-name>/cmb_msr/msr[0:31]
>   Date:		January 2024
> -KernelVersion	6.9
> +KernelVersion:	6.9
>   Contact:	Jinlong Mao (QUIC) <quic_jinlmao@quicinc.com>, Tao Zhang (QUIC) <quic_taozha@quicinc.com>
>   Description:
>   		(RW) Set/Get the MSR(mux select register) for the CMB subunit

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hi Jon,

> As I'll be preparing such patches for merge along this week, I'd
> like to know what do you prefer in terms of directories:
>
> 1. Keep it as-is;
> 2. have a separate library directory for Python modules
>    (scripts/lib?);
> 3. place python modules inside scripts/;
> 4. place python modules inside Documentation/sphinx (IMO a bad
>    idea);
> 5. something else

Honestly, I'm not sure.  I do feel that importing out of scripts/ is
inelegant at best; having a dedicated directory for modules meant to be
imported would be better.  So maybe scripts/lib?  Or lib/python, though
that might raise eyebrows elsewhere, dunno.  Pick something you like,
and we'll give that a try.

> Btw, I'm considering to also submit later a patchset similar to
> this one converting kernel-doc to Python. I already started writing
> something like that (written from the scratch, following as much
> as possible what we have today on Perl to avoid regressions).
>
> I would probably split the code into separate classes to make the code 
> more readable/maintainable (a base class, a class with rest output,
> another one with man output, and a few other helper classes).

I definitely approve of the idea - I've pondered doing such a thing, but
have never come close to finding the time.  It's probably worth looking
at the rewrite Markus did years ago as a starting point?

Thanks,

jon

Re: [RFC v2 05/38] docs: sphinx/kernel_abi: adjust coding style

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

So I was looking at applying the obvious fixes from this series, but ...

> Make pylint and flake8 happier with this module's coding style
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
>  Documentation/sphinx/kernel_abi.py | 28 +++++++++++++---------------
>  1 file changed, 13 insertions(+), 15 deletions(-)
>
> diff --git a/Documentation/sphinx/kernel_abi.py b/Documentation/sphinx/kernel_abi.py
> index 8401562cb5d9..a1887b50a900 100644
> --- a/Documentation/sphinx/kernel_abi.py
> +++ b/Documentation/sphinx/kernel_abi.py
> @@ -32,32 +32,30 @@ u"""
>  
>  """
>  
> -import codecs
>  import os
> -import subprocess

This breaks the build, since the module does indeed need the subprocess
module.

jon

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Mauro Carvalho Chehab]

Em Tue, 04 Feb 2025 10:12:22 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Hi Jon,  
> 
> > As I'll be preparing such patches for merge along this week, I'd
> > like to know what do you prefer in terms of directories:
> >
> > 1. Keep it as-is;
> > 2. have a separate library directory for Python modules
> >    (scripts/lib?);
> > 3. place python modules inside scripts/;
> > 4. place python modules inside Documentation/sphinx (IMO a bad
> >    idea);
> > 5. something else  
> 
> Honestly, I'm not sure.  I do feel that importing out of scripts/ is
> inelegant at best; having a dedicated directory for modules meant to be
> imported would be better.  So maybe scripts/lib?  Or lib/python, though
> that might raise eyebrows elsewhere, dunno.  Pick something you like,
> and we'll give that a try.

I would go for scripts/lib then.

> 
> > Btw, I'm considering to also submit later a patchset similar to
> > this one converting kernel-doc to Python. I already started writing
> > something like that (written from the scratch, following as much
> > as possible what we have today on Perl to avoid regressions).
> >
> > I would probably split the code into separate classes to make the code 
> > more readable/maintainable (a base class, a class with rest output,
> > another one with man output, and a few other helper classes).  
> 
> I definitely approve of the idea - I've pondered doing such a thing, but
> have never come close to finding the time.  It's probably worth looking
> at the rewrite Markus did years ago as a starting point?

I took a look on Markus work: it was licensed under GPL 3.0 and it was
written in 2016. There were several changes on kerneldoc since them,
including the addition of a regex that it is not compatible with
Python re[1]:

        $members =~ s/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/$2/gos;

This one use:

	- recursive patterns: ?1
	- atomic grouping (?>...)

Also, it is hard to map what he does with the existing script. I'm
opting to write a new script from scratch. I have already a version
that does ReST conversion (still missing man pages and filtering).
I'll send a RFC patch once I get it to the point it works properly.

[1] In the specific case of such regex, added on this commit:

	 50d7bd38c3aa ("stddef: Introduce struct_group() helper macro")

    I'm not sure why a very complex regex was used there. I'm currently 
    stuck on handling it properly in Python. An option would be do to:

	try:
		import regex as re
		has_regex_advanced_features = True
	except:
		import re
		has_regex_advanced_features = False

    And, if has_regex_advanced_features is false, use a simpler regex 
    that would work in Python or would just ignore struct_group
    expressions.

    Another option would be to re-implement such regexes without using
    such advanced patterns.

Thanks,
Mauro

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> I took a look on Markus work: it was licensed under GPL 3.0 and it was
> written in 2016. There were several changes on kerneldoc since them,
> including the addition of a regex that it is not compatible with
> Python re[1]:
>
>         $members =~ s/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/$2/gos;
>
> This one use:
>
> 	- recursive patterns: ?1
> 	- atomic grouping (?>...)
>
> Also, it is hard to map what he does with the existing script. I'm
> opting to write a new script from scratch.

That's fine, I just wanted to be sure you'd had a chance to look at
it... 

>     Another option would be to re-implement such regexes without using
>     such advanced patterns.

Seems like a preferred option if that can be done.  Banging one's head
against all those regexes is often the hardest part of dealing with that
script; anything that makes it simpler is welcome.

Thanks,

jon

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Mauro Carvalho Chehab]

Em Mon, 10 Feb 2025 07:40:02 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > I took a look on Markus work: it was licensed under GPL 3.0 and it was
> > written in 2016. There were several changes on kerneldoc since them,
> > including the addition of a regex that it is not compatible with
> > Python re[1]:
> >
> >         $members =~ s/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/$2/gos;
> >
> > This one use:
> >
> > 	- recursive patterns: ?1
> > 	- atomic grouping (?>...)
> >
> > Also, it is hard to map what he does with the existing script. I'm
> > opting to write a new script from scratch.  
> 
> That's fine, I just wanted to be sure you'd had a chance to look at
> it... 
> 
> >     Another option would be to re-implement such regexes without using
> >     such advanced patterns.  
> 
> Seems like a preferred option if that can be done.  Banging one's head
> against all those regexes is often the hardest part of dealing with that
> script; anything that makes it simpler is welcome.

Agreed. This one, in special, is very hard for me to understand, as I
never used recursive patterns or atomic grouping. The net result of
the struct_group*() handling is that it removes some parameters when
generating the function prototype. This is done using a complex logic
on two steps:

       # unwrap struct_group():
        # - first eat non-declaration parameters and rewrite for final match
        # - then remove macro, outer parens, and trailing semicolon
        $members =~ s/\bstruct_group\s*\(([^,]*,)/STRUCT_GROUP(/gos;
        $members =~ s/\bstruct_group_attr\s*\(([^,]*,){2}/STRUCT_GROUP(/gos;
        $members =~ s/\bstruct_group_tagged\s*\(([^,]*),([^,]*),/struct $1 $2; STRUCT_GROUP(/gos;
        $members =~ s/\b__struct_group\s*\(([^,]*,){3}/STRUCT_GROUP(/gos;
        $members =~ s/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/$2/gos;


The first step basically eliminates some members of the function. At the
places I checked, the second step was just removing parenthesis from the
macro (and the STRUCT_GROUP name).

I suspect that the same result could be done with a much simpler expression
like:

	$members =~ s/\bSTRUCT_GROUP\((.*)\)[^;]*;/$2/gos;

But maybe there are some corner cases that would cause such simpler
regex to fail.

-

On a side note, the "o" flag used there at kernel-doc is described
as[1]:

	"o  - pretend to optimize your code, but actually introduce bugs"

I wonder if we're reaching any issues on kernel docs due to that ;-)

[1] https://perldoc.perl.org/perlre::#Other-Modifiers

> 
> Thanks,
> 
> jon

Re: [RFC v2 18/38] docs: sphinx/kernel_abi: use AbiParser directly

[Mauro Carvalho Chehab]

Em Mon, 10 Feb 2025 17:03:54 +0100
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> Em Mon, 10 Feb 2025 07:40:02 -0700
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >   
> > > I took a look on Markus work: it was licensed under GPL 3.0 and it was
> > > written in 2016. There were several changes on kerneldoc since them,
> > > including the addition of a regex that it is not compatible with
> > > Python re[1]:
> > >
> > >         $members =~ s/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/$2/gos;
> > >
> > > This one use:
> > >
> > > 	- recursive patterns: ?1
> > > 	- atomic grouping (?>...)
> > >
> > > Also, it is hard to map what he does with the existing script. I'm
> > > opting to write a new script from scratch.    
> > 
> > That's fine, I just wanted to be sure you'd had a chance to look at
> > it... 
> >   
> > >     Another option would be to re-implement such regexes without using
> > >     such advanced patterns.    
> > 
> > Seems like a preferred option if that can be done.  Banging one's head
> > against all those regexes is often the hardest part of dealing with that
> > script; anything that makes it simpler is welcome.  
> 
> Agreed. This one, in special, is very hard for me to understand, as I
> never used recursive patterns or atomic grouping. The net result of
> the struct_group*() handling is that it removes some parameters when
> generating the function prototype. This is done using a complex logic
> on two steps:
> 
>        # unwrap struct_group():
>         # - first eat non-declaration parameters and rewrite for final match
>         # - then remove macro, outer parens, and trailing semicolon
>         $members =~ s/\bstruct_group\s*\(([^,]*,)/STRUCT_GROUP(/gos;
>         $members =~ s/\bstruct_group_attr\s*\(([^,]*,){2}/STRUCT_GROUP(/gos;
>         $members =~ s/\bstruct_group_tagged\s*\(([^,]*),([^,]*),/struct $1 $2; STRUCT_GROUP(/gos;
>         $members =~ s/\b__struct_group\s*\(([^,]*,){3}/STRUCT_GROUP(/gos;
>         $members =~ s/\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;/$2/gos;
> 
> 
> The first step basically eliminates some members of the function. At the
> places I checked, the second step was just removing parenthesis from the
> macro (and the STRUCT_GROUP name).
> 
> I suspect that the same result could be done with a much simpler expression
> like:
> 
> 	$members =~ s/\bSTRUCT_GROUP\((.*)\)[^;]*;/$2/gos;
> 
> But maybe there are some corner cases that would cause such simpler
> regex to fail.

A simpler regex didn't work.

Instead of using Python regex module and keep such complex regular
expressions, I opted to implement it on a completely different way.
See patch below.

Basically, I implemented a new class (NestedMatch) just to
handle patterns like:

	FOO(.*)

converting them into:

	.*

I expect that the new logic there would work properly for any
number of [], {} and () paired delimiters.

I ended writing such class from scratch, based on a suggestion from
stackoverflow.

Using such logic, replacing STRUCT_GROUP(.*) by .* is as simple
as:

        sub_nested_prefixes = [
            (re.compile(r'\bSTRUCT_GROUP'),  r'\1'),
        ]

        nested = NestedMatch()

        for search, sub in sub_nested_prefixes:
            members = nested.sub(search, sub, members)

This should probably help cleaning up other similar patterns.
Let's see. Anyway, I expect that this would make the C parsing
part easier to maintain.

For the first version, I'll use the new way only when I notice
discrepancies between Perl and Python versions.

Thanks,
Mauro

---

[PATCH] scripts/kernel-doc.py: properly handle struct_group macros

Handing nested parenthesis with regular expressions is not an
easy task. It is even harder with Python's re module, as it
has a limited subset of regular expressions, missing more
advanced features.

We might use instead Python regex module, but still the
regular expressions are very hard to understand. So, instead,
add a logic to properly match delimiters.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

diff --git a/scripts/kernel-doc.py b/scripts/kernel-doc.py
index 9b10fd05b6af..df824692c4e7 100755
--- a/scripts/kernel-doc.py
+++ b/scripts/kernel-doc.py
@@ -92,6 +92,138 @@ class Re:
     def group(self, num):
         return self.last_match.group(num)
 
+class NestedMatch:
+    """
+    Finding nested delimiters is hard with regular expressions. It is
+    even harder on Python with its normal re module, as there are several
+    advanced regular expressions that are missing.
+
+    This is the case of this pattern:
+
+            '\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;'
+
+    which is used to properly match open/close parenthesis of the
+    string search STRUCT_GROUP(),
+
+    Add a class that counts pairs of delimiters, using it to match and
+    replace nested expressions.
+
+    The original approach was suggested by:
+        https://stackoverflow.com/questions/5454322/python-how-to-match-nested-parentheses-with-regex
+
+    Although I re-implemented it to make it more generic and match 3 types
+    of delimiters. The logic checks if delimiters are paired. If not, it
+    will ignore the search string.
+    """
+
+    DELIMITER_PAIRS = {
+        '{': '}',
+        '(': ')',
+        '[': ']',
+    }
+
+    RE_DELIM = re.compile(r'[\{\}\[\]\(\)]')
+
+    def _search(self, regex, line):
+        """
+        Finds paired blocks.
+
+        If a given regex is not empty, the logic will first seek for its
+        position, and then handles the next delimiters afterwards
+
+        The suggestion of using finditer to match pairs came from:
+        https://stackoverflow.com/questions/5454322/python-how-to-match-nested-parentheses-with-regex
+        but I ended using a different implementation to align all three types
+        of delimiters and seek for an initial regular expression.
+
+        The algorithm seeks for open/close paired delimiters and place them
+        into a stack, yielding a start/stop position of each match  when the
+        stack is zeroed.
+
+        The algorithm shoud work fine for properly paired lines, but will
+        silently ignore end delimiters that preceeds an start delimiter.
+        This should be OK for kernel-doc parser, as unaligned delimiters
+        would cause compilation errors. So, we don't need to rise exceptions
+        to cover such issues.
+        """
+
+        stack = []
+
+        for match_re in regex.finditer(line):
+            offset = match_re.end()
+
+            for match in self.RE_DELIM.finditer(line[offset:]):
+                pos = match.start() + offset
+
+                d = line[pos]
+
+                if d in self.DELIMITER_PAIRS.keys():
+                    end = self.DELIMITER_PAIRS[d]
+
+                    stack.append(end)
+                    continue
+
+                # Does the end delimiter match what it is expected?
+                if stack and d == stack[-1]:
+                    stack.pop()
+
+                    if not stack:
+                        yield offset, pos + 1
+
+    def search(self, regex, line):
+        """
+        This is similar to re.search:
+
+        It matches a regex that it is followed by a delimiter,
+        returning occurrences only if all delimiters are paired.
+        """
+
+        for prev_pos, pos in self._search(regex, line):
+
+            yield line[prev_pos:pos]
+
+    def sub(self, regex, sub, line, count=0):
+        """
+        This is similar to re.sub:
+
+        It matches a regex that it is followed by a delimiter,
+        replacing occurrences only if all delimiters are paired.
+
+        if r'\1' is used, it works just like re: it places there the
+        matched paired data with the delimiter stripped.
+
+        If count is different than zero, it will replace at most count
+        items.
+        """
+        out = ""
+
+        cur_pos = 0
+        n = 0
+
+        for start, end in self._search(regex, line):
+            out += line[cur_pos:start]
+
+            # Value, ignoring start/end delimiters
+            value = line[start + 1:end - 1]
+
+            # replaces \1 at the sub string, if \1 is used there
+            new_sub = sub
+            new_sub = new_sub.replace(r'\1', value)
+
+            out += new_sub
+
+            cur_pos = end
+            n += 1
+
+            if count and count >= n:
+                break
+
+        # Append the remaining string
+        l = len(line)
+        out += line[cur_pos:l]
+
+        return out
+
 #
 # Regular expressions used to parse kernel-doc markups at KernelDoc class.
 #
@@ -667,19 +799,40 @@ class KernelDoc:
             # Unwrap struct_group() based on this definition:
             # __struct_group(TAG, NAME, ATTRS, MEMBERS...)
             # which has variants like: struct_group(NAME, MEMBERS...)
+            #
+            # Parsing them are done on two steps:
+            # 1. drop arguments that aren't struct names;
+            # 2. remove STRUCT_GROUP() ancillary macro.
+            #
+            # NOTE: the original logic which replaces STRUCT_GROUP(.*) by .*
+            # is incompatible with Python re, as it uses:
+            #
+            #   - a recursive pattern: (?1)
+            #   - an atomic grouping: (?>...)
+            #
+            # This is the original expression:
+            #   '\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;'
+            #
+            # I tried a simpler version: but it didn't work either:
+            #   \bSTRUCT_GROUP\(([^\)]+)\)[^;]*;
+            #
+            # As it doesn't properly match the end parenthesis.
+            #
+            # So, a better solution was crafted: there's now a NestedMatch
+            # class that ensures that delimiters after a search are properly
+            # matched. So, the implementation to drop STRUCT_GROUP() will be
+            # handled in separate.
 
             (Re(r'\bstruct_group\s*\(([^,]*,)', re.S),  r'STRUCT_GROUP('),
             (Re(r'\bstruct_group_attr\s*\(([^,]*,){2}', re.S),  r'STRUCT_GROUP('),
             (Re(r'\bstruct_group_tagged\s*\(([^,]*),([^,]*),', re.S),  r'struct \1 \2; STRUCT_GROUP('),
             (Re(r'\b__struct_group\s*\(([^,]*,){3}', re.S),  r'STRUCT_GROUP('),
 
-            # This is incompatible with Python re, as it uses:
-            #  recursive patterns ((?1)) and atomic grouping ((?>...)):
-            #   '\bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*;'
-            # Let's see if this works instead:
-            (Re(r'\bSTRUCT_GROUP\(([^\)]+)\)[^;]*;', re.S),  r'\1'),
-
             # Replace macros
+            #
+            # TODO: it is better to also move those to the NestedMatch logic,
+            # to endure that parenthesis will be properly matched.
+
             (Re(r'__ETHTOOL_DECLARE_LINK_MODE_MASK\s*\(([^\)]+)\)', re.S),  r'DECLARE_BITMAP(\1, __ETHTOOL_LINK_MODE_MASK_NBITS)'),
             (Re(r'DECLARE_PHY_INTERFACE_MASK\s*\(([^\)]+)\)', re.S),  r'DECLARE_BITMAP(\1, PHY_INTERFACE_MODE_MAX)'),
             (Re(r'DECLARE_BITMAP\s*\(' + args_pattern + r',\s*' + args_pattern + r'\)', re.S),  r'unsigned long \1[BITS_TO_LONGS(\2)]'),
@@ -691,9 +844,18 @@ class KernelDoc:
             (Re(r'DEFINE_DMA_UNMAP_LEN\s*\(' + args_pattern + r'\)', re.S),  r'__u32 \1'),
         ]
 
+        sub_nested_prefixes = [
+            (re.compile(r'\bSTRUCT_GROUP'),  r'\1'),
+        ]
+
         for search, sub in sub_prefixes:
             members = search.sub(sub, members)
 
+        nested = NestedMatch()
+
+        for search, sub in sub_nested_prefixes:
+            members = nested.sub(search, sub, members)
+
         # Keeps the original declaration as-is
         declaration = members
 

Re: [f2fs-dev] [RFC v2 00/38] Improve ABI documentation generation

[patchwork-bot+f2fs]

Hello:

This patch was applied to jaegeuk/f2fs.git (dev)
by Jonathan Corbet <corbet@lwn.net>:

On Tue, 28 Jan 2025 01:05:49 +0100 you wrote:
> Hi Jon/Greg,
> 
> That's the second version of my RFC patches meant to modenize the ABI
> parser that I wrote in Perl.
> 
> I originally started it due to some issues I noticed when searching for
> ABI symbols. While I could just go ahead and fix the already existing
> script, I noticed that the script maintainance didn't have much care over
> all those years, probably because it is easier to find Python programmers
> those days.
> 
> [...]

Here is the summary with links:
  - [f2fs-dev,RFC,v2,10/38] ABI: sysfs-fs-f2fs: fix date tags
    https://git.kernel.org/jaegeuk/f2fs/c/90800df0da78

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html