[PATCH v4 00/41] kernel-doc: make it parse new functions and structs

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

Hi Jon,

Rebased the changes on the top of docs-mw, and merged
the /4 patch series with fixes from me that ws pending
and contained a conflicting patch with this series.

Oh well, it means that this one increased its size
again...

The initial goal were to address some new warnings
at linux next due to some new macro annotations.

As the previous version, this one contains a new
CFunction class, with is just an alias for "NestedMatch",
meant to make easier to maintain transforms, which are
now on a separate file:

    tools/lib/python/kdoc/xforms_lists.py

With that, transform lists now use CFunction for macros
with a function prototype, using KernRe only the
remaining macros and part of the declarations we want to
strip (like extern).

So, for instance, the entire transform list for structs
is now:

    #: Transforms for structs and unions
    struct_xforms = [
        (CFunction("__attribute__"), ' '),
        (CFunction('__aligned'), ' '),
        (CFunction('__counted_by'), ' '),
        (CFunction('__counted_by_(le|be)'), ' '),
        (CFunction('__guarded_by'), ' '),
        (CFunction('__pt_guarded_by'), ' '),

        (KernRe(r'\s*__packed\s*', re.S), ' '),
        (KernRe(r'\s*CRYPTO_MINALIGN_ATTR', re.S), ' '),
        (KernRe(r'\s*__private', re.S), ' '),
        (KernRe(r'\s*__rcu', re.S), ' '),
        (KernRe(r'\s*____cacheline_aligned_in_smp', re.S), ' '),
        (KernRe(r'\s*____cacheline_aligned', re.S), ' '),

        (CFunction('__cacheline_group_(begin|end)'), ''),

        (CFunction('struct_group'), r'\2'),
        (CFunction('struct_group_attr'), r'\3'),
        (CFunction('struct_group_tagged'), r'struct \1 \2; \3'),
        (CFunction('__struct_group'), r'\4'),

        (CFunction('__ETHTOOL_DECLARE_LINK_MODE_MASK'), r'DECLARE_BITMAP(\1, __ETHTOOL_LINK_MODE_MASK_NBITS)'),
        (CFunction('DECLARE_PHY_INTERFACE_MASK',), r'DECLARE_BITMAP(\1, PHY_INTERFACE_MODE_MAX)'),
        (CFunction('DECLARE_BITMAP'), r'unsigned long \1[BITS_TO_LONGS(\2)]'),

        (CFunction('DECLARE_HASHTABLE'), r'unsigned long \1[1 << ((\2) - 1)]'),
        (CFunction('DECLARE_KFIFO'), r'\2 *\1'),
        (CFunction('DECLARE_KFIFO_PTR'), r'\2 *\1'),
        (CFunction('(?:__)?DECLARE_FLEX_ARRAY'), r'\1 \2[]'),
        (CFunction('DEFINE_DMA_UNMAP_ADDR'), r'dma_addr_t \1'),
        (CFunction('DEFINE_DMA_UNMAP_LEN'), r'__u32 \1'),
        (CFunction('VIRTIO_DECLARE_FEATURES'), r'union { u64 \1; u64 \1_array[VIRTIO_FEATURES_U64S]; }'),
    ]

As KernRe, CFunction and NestedMatch have a ".sub" method, a
single transforms table can have them all.

Due to the complexity of NestedMatch, I opted to write
unit tests to verify that the logic there is correct.

On this version, the module was almost entirely re-written
by hand, and the test suite has now 3 groups:

- TestDifferentReplacements: basic tests using several
  tests to try checkin border cases with NestedMatch.sub.
  They all use a single macro name;
- TestMultipleMacros: test different macros and conditions,
  also trying to get border cases;
- TestRealUsecases: as the name says, on this one I picked
  real test cases. I used an example given by Randy on
  some of them and placed a testset to try covering
  struct_group macros, which are quite complex.

Using it is as easy as running:

	$ tools/unittests/kdoc_xforms.py

And, if something goes wrong, -v will show the problem in unified diff[1]:

    $ tools/unittests/kdoc_xforms.py -k test_functions_with_acquires_and_releases -v

    test_functions_with_acquires_and_releases (kdoc_xforms.TestRealUsecases.test_functions_with_acquires_and_releases)
    Test guarded_by on vars from lib/stackdepot.c. ... FAIL
    
    ======================================================================
    FAIL: test_functions_with_acquires_and_releases (kdoc_xforms.TestRealUsecases.test_functions_with_acquires_and_releases)
    Test guarded_by on vars from lib/stackdepot.c.
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/new_devel/docs/tools/unittests/kdoc_xforms.py", line 649, in test_functions_with_acquires_and_releases
        self.assertLogicallyEqual(result, expected)
        ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
      File "/new_devel/docs/tools/unittests/kdoc_xforms.py", line 56, in assertLogicallyEqual
        self.assertEqual(a, b)
        ~~~~~~~~~~~~~~~~^^^^^^
    AssertionError: 'bool[477 chars]\n\n acpi_cpu_flags acpi_os_acquire_lock(acpi_[90 chars]sed)' != 'bool[477 chars]\n\n /* THIS IS UNEXPECTED. TEST WILL FAIL HER[140 chars]sed)'
      bool prepare_report_consumer(unsigned long *flags,
       const struct access_info *ai,
       struct other_info *other_info);
      
       int tcp_sigpool_start(unsigned int id, struct tcp_sigpool *c);
      
       bool undo_report_consumer(unsigned long *flags,
       const struct access_info *ai,
       struct other_info *other_info);
      
       void debugfs_enter_cancellation(struct file *file,
       struct debugfs_cancellation *c);
      
       void debugfs_leave_cancellation(struct file *file,
       struct debugfs_cancellation *c);
      
    +  /* THIS IS UNEXPECTED. TEST WILL FAIL HERE */
    + 
       acpi_cpu_flags acpi_os_acquire_lock(acpi_spinlock lockp);
      
       void acpi_os_release_lock(acpi_spinlock lockp,
       acpi_cpu_flags not_used)
    
    
    ----------------------------------------------------------------------
    Ran 1 test in 0.001s
    
    FAILED (failures=1)

[1] This error was caused when I applied this patch:

    --- a/tools/unittests/kdoc_xforms.py
    +++ b/tools/unittests/kdoc_xforms.py
    @@ -637,6 +637,8 @@ class TestRealUsecases(TestCaseDiff):
                 void debugfs_leave_cancellation(struct file *file,
                                                 struct debugfs_cancellation *c);
 
    +            /* THIS IS UNEXPECTED. TEST WILL FAIL HERE */
    +
                 acpi_cpu_flags acpi_os_acquire_lock(acpi_spinlock lockp);
 
                 void acpi_os_release_lock(acpi_spinlock lockp,



---
v4:
- minor cleanups at the description on the first 30 patches;
- picked paches from this series:
    https://lore.kernel.org/linux-doc/cover.1769500383.git.mchehab+huawei@kernel.org/
- added patches to make easier for kernel-doc to be used on
  other projects (QEMU currently uses it)
- did some extra fixes at NestedMatch for some corner cases;
- added patches to improve man pages output;
- unittests were revamped.

v3:
- improved the unittest helper to allow adding in the future
  a runner to create a test suite directly;
- added unittest to tools/python library documentation;
- improved comments at the new modules;
- did several cleanups at the new logic;
- added a fix for NestedMatch not remove ";" at the end,
  mimicing the behavior of KernRe;
- moved transforms to a separate module;
- replaced all regexes to parse macros with the new CFunction
  alias for NestedMatch.

v2:
- added 10 new patches adding support at NestedMatch
  to properly group and replace arguments with \1, \2, ...

Mauro Carvalho Chehab (39):
  docs: kdoc_re: add support for groups()
  docs: kdoc_re: don't go past the end of a line
  docs: kdoc_parser: move var transformers to the beginning
  docs: kdoc_parser: don't mangle with function defines
  docs: kdoc_parser: add functions support for NestedMatch
  docs: kdoc_parser: use NestedMatch to handle __attribute__ on
    functions
  docs: kdoc_parser: fix variable regexes to work with size_t
  docs: kdoc_parser: fix the default_value logic for variables
  docs: kdoc_parser: add some debug for variable parsing
  docs: kdoc_parser: don't exclude defaults from prototype
  docs: kdoc_parser: fix parser to support multi-word types
  docs: kdoc_parser: add support for LIST_HEAD
  docs: kdoc_re: properly handle strings and escape chars on it
  docs: kdoc_re: better show KernRe() at documentation
  docs: kdoc_re: don't recompile NestedMatch regex every time
  docs: kdoc_re: Change NestedMath args replacement to \0
  docs: kdoc_re: make NestedMatch use KernRe
  docs: kdoc_re: add support on NestedMatch for argument replacement
  docs: kdoc_parser: better handle struct_group macros
  docs: kdoc_re: fix a parse bug on struct page_pool_params
  docs: kdoc_re: add a helper class to declare C function matches
  docs: kdoc_parser: use the new CFunction class
  docs: kdoc_parser: minimize differences with struct_group_tagged
  docs: kdoc_parser: move transform lists to a separate file
  docs: kdoc_re: don't remove the trailing ";" with NestedMatch
  docs: kdoc_re: prevent adding whitespaces on sub replacements
  docs: xforms_lists.py: use CFuntion to handle all function macros
  docs: kdoc_files: allows the caller to use a different xforms class
  docs: kdoc_re: Fix NestedMatch.sub() which causes PDF builds to break
  docs: sphinx-build-wrapper: allow -v override -q
  tools: sphinx-build-wrapper: improve its help message
  docs: kdoc_files: document KernelFiles() ABI
  docs: kdoc_output: add optional args to ManOutput class
  docs: sphinx-build-wrapper: better handle troff .TH markups
  docs: kdoc_output: use a more standard order for .TH on man pages
  docs: kdoc_output: describe the class init parameters
  docs: kdoc_output: pick a better default for modulename
  docs: python: add helpers to run unit tests
  tools: unittests: add tests for kernel-doc NestedMatch and KernRe

Randy Dunlap (2):
  docs: kdoc_parser: ignore context analysis and lock attributes
  docs: kdoc_parser: handle struct member macro
    VIRTIO_DECLARE_FEATURES(name)

 Documentation/tools/kdoc_parser.rst   |   8 +
 Documentation/tools/python.rst        |   2 +
 Documentation/tools/unittest.rst      |  24 +
 tools/docs/kernel-doc                 |   1 -
 tools/docs/sphinx-build-wrapper       |  50 +-
 tools/lib/python/kdoc/kdoc_files.py   |  54 ++-
 tools/lib/python/kdoc/kdoc_output.py  |  73 ++-
 tools/lib/python/kdoc/kdoc_parser.py  | 182 ++-----
 tools/lib/python/kdoc/kdoc_re.py      | 242 +++++++---
 tools/lib/python/kdoc/xforms_lists.py | 108 +++++
 tools/lib/python/unittest_helper.py   | 348 ++++++++++++++
 tools/unittests/kdoc_xforms.py        | 654 ++++++++++++++++++++++++++
 12 files changed, 1515 insertions(+), 231 deletions(-)
 create mode 100644 Documentation/tools/unittest.rst
 create mode 100644 tools/lib/python/kdoc/xforms_lists.py
 create mode 100755 tools/lib/python/unittest_helper.py
 create mode 100755 tools/unittests/kdoc_xforms.py

-- 
2.52.0