[PATCH net-next v1 0/3] doc: netlink: Add hyperlinks to generated docs

[PATCH net-next v1 0/3] doc: netlink: Add hyperlinks to generated docs

[Donald Hunter]

Extemd ynl-gen-rst to generate hyperlinks to definitions, attribute sets
and sub-messages from all the places that reference them.

Donald Hunter (3):
  doc: netlink: Change generated docs to limit TOC to depth 3
  doc: netlink: Add hyperlinks to generated Netlink docs
  doc: netlink: Update tc spec with missing definitions

 Documentation/netlink/specs/tc.yaml | 51 +++++++++++++++++++++++++++++
 tools/net/ynl/ynl-gen-rst.py        | 46 +++++++++++++++++++-------
 2 files changed, 85 insertions(+), 12 deletions(-)

-- 
2.44.0

[PATCH net-next v1 1/3] doc: netlink: Change generated docs to limit TOC to depth 3

[Donald Hunter]

The tables of contents in the generated Netlink docs include individual
attribute definitions. This can make the contents exceedingly long and
repeats a lot of what is on the rest of the pages. See for example:

https://docs.kernel.org/networking/netlink_spec/tc.html

Add a depth limit to the contents directive in generated .rst files to
limit the contents depth to 3 levels. This reduces the contents to:

 - Family
   - Summary
   - Operations
     - op-one
     - op-two
     - ...
   - Definitions
     - struct-one
     - struct-two
     - enum-one
     - ...
   - Attribute sets
     - attrs-one
     - attrs-two
     - ...

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 tools/net/ynl/ynl-gen-rst.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/net/ynl/ynl-gen-rst.py b/tools/net/ynl/ynl-gen-rst.py
index 927407b3efb3..5825a8b3bfb4 100755
--- a/tools/net/ynl/ynl-gen-rst.py
+++ b/tools/net/ynl/ynl-gen-rst.py
@@ -291,7 +291,7 @@ def parse_yaml(obj: Dict[str, Any]) -> str:
 
     title = f"Family ``{obj['name']}`` netlink specification"
     lines.append(rst_title(title))
-    lines.append(rst_paragraph(".. contents::\n"))
+    lines.append(rst_paragraph(".. contents:: :depth: 3\n"))
 
     if "doc" in obj:
         lines.append(rst_subtitle("Summary"))
-- 
2.44.0

[PATCH net-next v1 2/3] doc: netlink: Add hyperlinks to generated Netlink docs

[Donald Hunter]

Update ynl-gen-rst to generate hyperlinks to definitions, attribute
sets and sub-messages from all the places that reference them.

Note that there is a single label namespace for all of the kernel docs.
Hyperlinks within a single netlink doc need to be qualified by the
family name to avoid collisions.

The label format is 'family-type-name' which gives, for example,
'rt-link-attribute-set-link-attrs' as the link id.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 tools/net/ynl/ynl-gen-rst.py | 44 +++++++++++++++++++++++++++---------
 1 file changed, 33 insertions(+), 11 deletions(-)

diff --git a/tools/net/ynl/ynl-gen-rst.py b/tools/net/ynl/ynl-gen-rst.py
index 5825a8b3bfb4..4be931c9bdbf 100755
--- a/tools/net/ynl/ynl-gen-rst.py
+++ b/tools/net/ynl/ynl-gen-rst.py
@@ -82,9 +82,9 @@ def rst_subsubsection(title: str) -> str:
     return f"{title}\n" + "~" * len(title)
 
 
-def rst_section(title: str) -> str:
+def rst_section(prefix: str, title: str) -> str:
     """Add a section to the document"""
-    return f"\n{title}\n" + "=" * len(title)
+    return f".. _{family}-{prefix}-{title}:\n\n{title}\n" + "=" * len(title)
 
 
 def rst_subtitle(title: str) -> str:
@@ -101,6 +101,16 @@ def rst_list_inline(list_: List[str], level: int = 0) -> str:
     """Format a list using inlines"""
     return headroom(level) + "[" + ", ".join(inline(i) for i in list_) + "]"
 
+def rst_ref(prefix: str, name: str) -> str:
+    """Add a hyperlink to the document"""
+    mappings = {'enum': 'definition',
+                'fixed-header': 'definition',
+                'nested-attributes': 'attribute-set',
+                'struct': 'definition'}
+    if prefix in mappings:
+        prefix = mappings[prefix]
+    return f":ref:`{family}-{prefix}-{name}`"
+
 
 def rst_header() -> str:
     """The headers for all the auto generated RST files"""
@@ -162,17 +172,21 @@ def parse_do_attributes(attrs: Dict[str, Any], level: int = 0) -> str:
 def parse_operations(operations: List[Dict[str, Any]]) -> str:
     """Parse operations block"""
     preprocessed = ["name", "doc", "title", "do", "dump"]
+    linkable = ["fixed-header", "attribute-set"]
     lines = []
 
     for operation in operations:
-        lines.append(rst_section(operation["name"]))
+        lines.append(rst_section('operation', operation["name"]))
         lines.append(rst_paragraph(sanitize(operation["doc"])) + "\n")
 
         for key in operation.keys():
             if key in preprocessed:
                 # Skip the special fields
                 continue
-            lines.append(rst_fields(key, operation[key], 0))
+            value = operation[key]
+            if key in linkable:
+                value = rst_ref(key, value)
+            lines.append(rst_fields(key, value, 0))
 
         if "do" in operation:
             lines.append(rst_paragraph(":do:", 0))
@@ -219,7 +233,7 @@ def parse_definitions(defs: Dict[str, Any]) -> str:
     lines = []
 
     for definition in defs:
-        lines.append(rst_section(definition["name"]))
+        lines.append(rst_section('definition', definition["name"]))
         for k in definition.keys():
             if k in preprocessed + ignored:
                 continue
@@ -240,11 +254,12 @@ def parse_definitions(defs: Dict[str, Any]) -> str:
 def parse_attr_sets(entries: List[Dict[str, Any]]) -> str:
     """Parse attribute from attribute-set"""
     preprocessed = ["name", "type"]
+    linkable = ["enum", "nested-attributes", "struct", "sub-message"]
     ignored = ["checks"]
     lines = []
 
     for entry in entries:
-        lines.append(rst_section(entry["name"]))
+        lines.append(rst_section('attribute-set', entry["name"]))
         for attr in entry["attributes"]:
             type_ = attr.get("type")
             attr_line = attr["name"]
@@ -257,7 +272,11 @@ def parse_attr_sets(entries: List[Dict[str, Any]]) -> str:
             for k in attr.keys():
                 if k in preprocessed + ignored:
                     continue
-                lines.append(rst_fields(k, sanitize(attr[k]), 0))
+                if k in linkable:
+                    value = rst_ref(k, attr[k])
+                else:
+                    value = sanitize(attr[k])
+                lines.append(rst_fields(k, value, 0))
             lines.append("\n")
 
     return "\n".join(lines)
@@ -268,14 +287,14 @@ def parse_sub_messages(entries: List[Dict[str, Any]]) -> str:
     lines = []
 
     for entry in entries:
-        lines.append(rst_section(entry["name"]))
+        lines.append(rst_section('sub-message', entry["name"]))
         for fmt in entry["formats"]:
             value = fmt["value"]
 
             lines.append(rst_bullet(bold(value)))
             for attr in ['fixed-header', 'attribute-set']:
                 if attr in fmt:
-                    lines.append(rst_fields(attr, fmt[attr], 1))
+                    lines.append(rst_fields(attr, rst_ref(attr, fmt[attr]), 1))
             lines.append("\n")
 
     return "\n".join(lines)
@@ -289,7 +308,11 @@ def parse_yaml(obj: Dict[str, Any]) -> str:
 
     lines.append(rst_header())
 
-    title = f"Family ``{obj['name']}`` netlink specification"
+    # Save the family for use in ref labels
+    global family
+    family = obj['name']
+
+    title = f"Family ``{family}`` netlink specification"
     lines.append(rst_title(title))
     lines.append(rst_paragraph(".. contents:: :depth: 3\n"))
 
@@ -398,7 +421,6 @@ def generate_main_index_rst(output: str) -> None:
     logging.debug("Writing an index file at %s", output)
     write_to_rstfile("".join(lines), output)
 
-
 def main() -> None:
     """Main function that reads the YAML files and generates the RST files"""
 
-- 
2.44.0

[PATCH net-next v1 3/3] doc: netlink: Update tc spec with missing definitions

[Donald Hunter]

The tc spec referenced tc-u32-mark and tc-act-police-attrs but did not
define them. The missing definitions were discovered when building the
docs with generated hyperlinks because the hyperlink target labels were
missing.

Add definitions for tc-u32-mark and tc-act-police-attrs.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/netlink/specs/tc.yaml | 51 +++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/Documentation/netlink/specs/tc.yaml b/Documentation/netlink/specs/tc.yaml
index 324fa182cd14..6068c105c5ee 100644
--- a/Documentation/netlink/specs/tc.yaml
+++ b/Documentation/netlink/specs/tc.yaml
@@ -1099,6 +1099,19 @@ definitions:
       -
         name: offmask
         type: s32
+  -
+    name: tc-u32-mark
+    type: struct
+    members:
+      -
+        name: val
+        type: u32
+      -
+        name: mask
+        type: u32
+      -
+        name: success
+        type: u32
   -
     name: tc-u32-sel
     type: struct
@@ -1774,6 +1787,44 @@ attribute-sets:
       -
         name: key-ex
         type: binary
+  -
+    name: tc-act-police-attrs
+    attributes:
+      -
+        name: tbf
+        type: binary
+        struct: tc-police
+      -
+        name: rate
+        type: binary # TODO
+      -
+        name: peakrate
+        type: binary # TODO
+      -
+        name: avrate
+        type: u32
+      -
+        name: result
+        type: u32
+      -
+        name: tm
+        type: binary
+        struct: tcf-t
+      -
+        name: pad
+        type: pad
+      -
+        name: rate64
+        type: u64
+      -
+        name: peakrate64
+        type: u64
+      -
+        name: pktrate64
+        type: u64
+      -
+        name: pktburst64
+        type: u64
   -
     name: tc-act-simple-attrs
     attributes:
-- 
2.44.0

Re: [PATCH net-next v1 2/3] doc: netlink: Add hyperlinks to generated Netlink docs

[Breno Leitao]

Hi Donald,

On Tue, Mar 26, 2024 at 08:13:10PM +0000, Donald Hunter wrote:
> Update ynl-gen-rst to generate hyperlinks to definitions, attribute
> sets and sub-messages from all the places that reference them.
> 
> Note that there is a single label namespace for all of the kernel docs.
> Hyperlinks within a single netlink doc need to be qualified by the
> family name to avoid collisions.
> 
> The label format is 'family-type-name' which gives, for example,
> 'rt-link-attribute-set-link-attrs' as the link id.
> 
> Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
> ---
>  tools/net/ynl/ynl-gen-rst.py | 44 +++++++++++++++++++++++++++---------
>  1 file changed, 33 insertions(+), 11 deletions(-)
> 
> diff --git a/tools/net/ynl/ynl-gen-rst.py b/tools/net/ynl/ynl-gen-rst.py
> index 5825a8b3bfb4..4be931c9bdbf 100755
> --- a/tools/net/ynl/ynl-gen-rst.py
> +++ b/tools/net/ynl/ynl-gen-rst.py
> @@ -82,9 +82,9 @@ def rst_subsubsection(title: str) -> str:
>      return f"{title}\n" + "~" * len(title)
>  
>  
> -def rst_section(title: str) -> str:
> +def rst_section(prefix: str, title: str) -> str:
>      """Add a section to the document"""
> -    return f"\n{title}\n" + "=" * len(title)
> +    return f".. _{family}-{prefix}-{title}:\n\n{title}\n" + "=" * len(title)

Where is 'family' variable set? Is this a global variable somewhere?

Re: [PATCH net-next v1 1/3] doc: netlink: Change generated docs to limit TOC to depth 3

[Breno Leitao]

On Tue, Mar 26, 2024 at 08:13:09PM +0000, Donald Hunter wrote:
> The tables of contents in the generated Netlink docs include individual
> attribute definitions. This can make the contents exceedingly long and
> repeats a lot of what is on the rest of the pages. See for example:
> 
> https://docs.kernel.org/networking/netlink_spec/tc.html
> 
> Add a depth limit to the contents directive in generated .rst files to
> limit the contents depth to 3 levels. This reduces the contents to:
> 
>  - Family
>    - Summary
>    - Operations
>      - op-one
>      - op-two
>      - ...
>    - Definitions
>      - struct-one
>      - struct-two
>      - enum-one
>      - ...
>    - Attribute sets
>      - attrs-one
>      - attrs-two
>      - ...
> 
> Signed-off-by: Donald Hunter <donald.hunter@gmail.com>

Reviewed-by: Breno Leitao <leitao@debian.org>

Re: [PATCH net-next v1 2/3] doc: netlink: Add hyperlinks to generated Netlink docs

[Donald Hunter]

On Thu, 28 Mar 2024 at 15:00, Breno Leitao <leitao@debian.org> wrote:
>
> Hi Donald,
>
> On Tue, Mar 26, 2024 at 08:13:10PM +0000, Donald Hunter wrote:
> > Update ynl-gen-rst to generate hyperlinks to definitions, attribute
> > sets and sub-messages from all the places that reference them.
> >
> > Note that there is a single label namespace for all of the kernel docs.
> > Hyperlinks within a single netlink doc need to be qualified by the
> > family name to avoid collisions.
> >
> > The label format is 'family-type-name' which gives, for example,
> > 'rt-link-attribute-set-link-attrs' as the link id.
> >
> > Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
> > ---
> >  tools/net/ynl/ynl-gen-rst.py | 44 +++++++++++++++++++++++++++---------
> >  1 file changed, 33 insertions(+), 11 deletions(-)
> >
> > diff --git a/tools/net/ynl/ynl-gen-rst.py b/tools/net/ynl/ynl-gen-rst.py
> > index 5825a8b3bfb4..4be931c9bdbf 100755
> > --- a/tools/net/ynl/ynl-gen-rst.py
> > +++ b/tools/net/ynl/ynl-gen-rst.py
> > @@ -82,9 +82,9 @@ def rst_subsubsection(title: str) -> str:
> >      return f"{title}\n" + "~" * len(title)
> >
> >
> > -def rst_section(title: str) -> str:
> > +def rst_section(prefix: str, title: str) -> str:
> >      """Add a section to the document"""
> > -    return f"\n{title}\n" + "=" * len(title)
> > +    return f".. _{family}-{prefix}-{title}:\n\n{title}\n" + "=" * len(title)
>
> Where is 'family' variable set? Is this a global variable somewhere?

Yes, here in parse_yaml(). I realise it's a bit of a hack but would
like to clean this up as part of switching to using ynl/lib/nlspec.py
for reading the specs, in a separate patchset.

-    title = f"Family ``{obj['name']}`` netlink specification"
+    # Save the family for use in ref labels
+    global family
+    family = obj['name']
+
+    title = f"Family ``{family}`` netlink specification"

Re: [PATCH net-next v1 2/3] doc: netlink: Add hyperlinks to generated Netlink docs

[Breno Leitao]

On Thu, Mar 28, 2024 at 03:54:09PM +0000, Donald Hunter wrote:

> > > -def rst_section(title: str) -> str:
> > > +def rst_section(prefix: str, title: str) -> str:
> > >      """Add a section to the document"""
> > > -    return f"\n{title}\n" + "=" * len(title)
> > > +    return f".. _{family}-{prefix}-{title}:\n\n{title}\n" + "=" * len(title)
> >
> > Where is 'family' variable set? Is this a global variable somewhere?
> 
> Yes, here in parse_yaml(). I realise it's a bit of a hack but would
> like to clean this up as part of switching to using ynl/lib/nlspec.py
> for reading the specs, in a separate patchset.

Thanks. Is it worth adding a hack that would be removed later, other
than going straight to the final solution?

> -    title = f"Family ``{obj['name']}`` netlink specification"
> +    # Save the family for use in ref labels
> +    global family

Is it hard to pass this variable by without having to use a global
variable?

Re: [PATCH net-next v1 2/3] doc: netlink: Add hyperlinks to generated Netlink docs

[Jakub Kicinski]

On Tue, 26 Mar 2024 20:13:10 +0000 Donald Hunter wrote:
>      return headroom(level) + "[" + ", ".join(inline(i) for i in list_) + "]"
>  
> +def rst_ref(prefix: str, name: str) -> str:

I think python coding guidelines call for 2 empty lines between
functions? There's another place where this is violated in the patch.

FWIW I also feel like the using the global directly is a bit too hacky.
Dunno how much work it'd be to pass it in, but if it's a lot let's at
least define it at the start of the file and always have "global family"
before the use?
-- 
pw-bot: cr

Re: [PATCH net-next v1 2/3] doc: netlink: Add hyperlinks to generated Netlink docs

[Donald Hunter]

Jakub Kicinski <kuba@kernel.org> writes:

> On Tue, 26 Mar 2024 20:13:10 +0000 Donald Hunter wrote:
>>      return headroom(level) + "[" + ", ".join(inline(i) for i in list_) + "]"
>>  
>> +def rst_ref(prefix: str, name: str) -> str:
>
> I think python coding guidelines call for 2 empty lines between
> functions? There's another place where this is violated in the patch.

Good catch. I'll fix in v2.

> FWIW I also feel like the using the global directly is a bit too hacky.
> Dunno how much work it'd be to pass it in, but if it's a lot let's at
> least define it at the start of the file and always have "global family"
> before the use?

I will just bite the bullet and pass a parameter in everywhere for v2.