[PATCH net-next v1 0/6] tools/net/ynl: Add 'sub-message' support to ynl

[PATCH net-next v1 0/6] tools/net/ynl: Add 'sub-message' support to ynl

[Donald Hunter]

This patchset adds a 'sub-message' attribute type to the netlink-raw
schema and implements it in ynl. This provides support for kind-specific
options attributes as used in rt_link and tc raw netlink families.

A description of the new 'sub-message' attribute type and the
corresponding sub-message definitions is provided in patch 2.

The patchset includes updates to the rt_link spec and a new tc spec that
make use of the new 'sub-message' attribute type.

Patch 1 adds missing scalar types to the netlink-raw schema
Patch 2 and 3 add sub-message support to the schema and ynl
Patch 4 adds binary and pad support to structs in netlink-raw
Patches 5 and 6 contain specs that use the sub-message attribute type

Donald Hunter (6):
  doc/netlink: Add bitfield32, s8, s16 to the netlink-raw schema
  doc/netlink: Add sub-message support to netlink-raw
  tools/net/ynl: Add 'sub-message' attribute decoding to ynl
  tools/net/ynl: Add binary and pad support to structs for tc
  doc/netlink/specs: add sub-message type to rt_link family
  doc/netlink/specs: Add a spec for tc

 Documentation/netlink/netlink-raw.yaml   |   56 +-
 Documentation/netlink/specs/rt_link.yaml |  273 ++-
 Documentation/netlink/specs/tc.yaml      | 2008 ++++++++++++++++++++++
 tools/net/ynl/lib/nlspec.py              |   57 +-
 tools/net/ynl/lib/ynl.py                 |   84 +-
 5 files changed, 2452 insertions(+), 26 deletions(-)
 create mode 100644 Documentation/netlink/specs/tc.yaml

-- 
2.42.0

[PATCH net-next v1 1/6] doc/netlink: Add bitfield32, s8, s16 to the netlink-raw schema

[Donald Hunter]

The netlink-raw schema was not updated when bitfield32 was added
to the genetlink-legacy schema. It is needed for rtnetlink families.

s8 and s16 were also missing.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/netlink/netlink-raw.yaml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Documentation/netlink/netlink-raw.yaml b/Documentation/netlink/netlink-raw.yaml
index 775cce8c548a..ad5395040765 100644
--- a/Documentation/netlink/netlink-raw.yaml
+++ b/Documentation/netlink/netlink-raw.yaml
@@ -200,7 +200,8 @@ properties:
                 type: string
               type: &attr-type
                 description: The netlink attribute type
-                enum: [ unused, pad, flag, binary, u8, u16, u32, u64, s32, s64,
+                enum: [ unused, pad, flag, binary, bitfield32,
+                        u8, u16, u32, u64, s8, s16, s32, s64,
                         string, nest, array-nest, nest-type-value ]
               doc:
                 description: Documentation of the attribute.
-- 
2.42.0

[PATCH net-next v1 2/6] doc/netlink: Add sub-message support to netlink-raw

[Donald Hunter]

Add a 'sub-message' attribute type with a selector that supports
polymorphic attribute formats for raw netlink families like tc.

A sub-message attribute uses the value of another attribute as a
selector key to choose the right sub-message format. For example if the
following attribute has already been decoded:

  { "kind": "gre" }

and we encounter the following attribute spec:

  -
    name: data
    type: sub-message
    sub-message: linkinfo-data-msg
    selector: kind

Then we look for a sub-message definition called 'linkinfo-data-msg' and
use the value of the 'kind' attribute i.e. 'gre' as the key to choose
the correct format for the sub-message:

  sub-messages:
    name: linkinfo-data-msg
    formats:
      -
        value: bridge
        attribute-set: linkinfo-bridge-attrs
      -
        value: gre
        attribute-set: linkinfo-gre-attrs
      -
        value: geneve
        attribute-set: linkinfo-geneve-attrs

This would decode the attribute value as a sub-message with the
attribute-set called 'linkinfo-gre-attrs' as the attribute space.

A sub-message can have an optional 'fixed-header' followed by zero or
more attributes from an attribute-set. For example the following
'tc-options-msg' sub-message defines message formats that use a mixture
of fixed-header, attribute-set or both together:

  sub-messages:
    -
      name: tc-options-msg
      formats:
        -
          value: bfifo
          fixed-header: tc-fifo-qopt
        -
          value: cake
          attribute-set: tc-cake-attrs
        -
          value: netem
          fixed-header: tc-netem-qopt
          attribute-set: tc-netem-attrs

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/netlink/netlink-raw.yaml | 51 +++++++++++++++++++++++++-
 1 file changed, 50 insertions(+), 1 deletion(-)

diff --git a/Documentation/netlink/netlink-raw.yaml b/Documentation/netlink/netlink-raw.yaml
index ad5395040765..26203282422f 100644
--- a/Documentation/netlink/netlink-raw.yaml
+++ b/Documentation/netlink/netlink-raw.yaml
@@ -202,7 +202,8 @@ properties:
                 description: The netlink attribute type
                 enum: [ unused, pad, flag, binary, bitfield32,
                         u8, u16, u32, u64, s8, s16, s32, s64,
-                        string, nest, array-nest, nest-type-value ]
+                        string, nest, array-nest, nest-type-value,
+                        sub-message ]
               doc:
                 description: Documentation of the attribute.
                 type: string
@@ -261,6 +262,17 @@ properties:
                 description: Name of the struct type used for the attribute.
                 type: string
               # End genetlink-legacy
+              # Start netlink-raw
+              sub-message:
+                description:
+                  Name of the sub-message definition to use for the attribute.
+                type: string
+              selector:
+                description:
+                  Name of the attribute to use for dynamic selection of sub-message
+                  format specifier.
+                type: string
+              # End netlink-raw
 
       # Make sure name-prefix does not appear in subsets (subsets inherit naming)
       dependencies:
@@ -283,6 +295,43 @@ properties:
             items:
               required: [ type ]
 
+  # Start netlink-raw
+  sub-messages:
+    description: Definition of sub message attributes
+    type: array
+    items:
+      type: object
+      additionalProperties: False
+      required: [ name, formats ]
+      properties:
+        name:
+          description: Name of the sub-message definition
+          type: string
+        formats:
+          description: Dynamically selected format specifiers
+          type: array
+          items:
+            type: object
+            additionalProperties: False
+            required: [ value ]
+            properties:
+              value:
+                description:
+                  Value to match for dynamic selection of sub-message format
+                  specifier.
+                type: string
+              fixed-header:
+                description:
+                  Name of the struct definition to use as the fixed header
+                  for the sub message.
+                type: string
+              attribute-set:
+                description:
+                  Name of the attribute space from which to resolve attributes
+                  in the sub message.
+                type: string
+  # End netlink-raw
+
   operations:
     description: Operations supported by the protocol.
     type: object
-- 
2.42.0

Re: [PATCH net-next v1 0/6] tools/net/ynl: Add 'sub-message' support to ynl

[Donald Hunter]

Looks like I'll need to resend because I got send failures.

[PATCH net-next v1 2/6] doc/netlink: Add sub-message support to netlink-raw

[Donald Hunter]

Add a 'sub-message' attribute type with a selector that supports
polymorphic attribute formats for raw netlink families like tc.

A sub-message attribute uses the value of another attribute as a
selector key to choose the right sub-message format. For example if the
following attribute has already been decoded:

  { "kind": "gre" }

and we encounter the following attribute spec:

  -
    name: data
    type: sub-message
    sub-message: linkinfo-data-msg
    selector: kind

Then we look for a sub-message definition called 'linkinfo-data-msg' and
use the value of the 'kind' attribute i.e. 'gre' as the key to choose
the correct format for the sub-message:

  sub-messages:
    name: linkinfo-data-msg
    formats:
      -
        value: bridge
        attribute-set: linkinfo-bridge-attrs
      -
        value: gre
        attribute-set: linkinfo-gre-attrs
      -
        value: geneve
        attribute-set: linkinfo-geneve-attrs

This would decode the attribute value as a sub-message with the
attribute-set called 'linkinfo-gre-attrs' as the attribute space.

A sub-message can have an optional 'fixed-header' followed by zero or
more attributes from an attribute-set. For example the following
'tc-options-msg' sub-message defines message formats that use a mixture
of fixed-header, attribute-set or both together:

  sub-messages:
    -
      name: tc-options-msg
      formats:
        -
          value: bfifo
          fixed-header: tc-fifo-qopt
        -
          value: cake
          attribute-set: tc-cake-attrs
        -
          value: netem
          fixed-header: tc-netem-qopt
          attribute-set: tc-netem-attrs

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/netlink/netlink-raw.yaml | 51 +++++++++++++++++++++++++-
 1 file changed, 50 insertions(+), 1 deletion(-)

diff --git a/Documentation/netlink/netlink-raw.yaml b/Documentation/netlink/netlink-raw.yaml
index ad5395040765..26203282422f 100644
--- a/Documentation/netlink/netlink-raw.yaml
+++ b/Documentation/netlink/netlink-raw.yaml
@@ -202,7 +202,8 @@ properties:
                 description: The netlink attribute type
                 enum: [ unused, pad, flag, binary, bitfield32,
                         u8, u16, u32, u64, s8, s16, s32, s64,
-                        string, nest, array-nest, nest-type-value ]
+                        string, nest, array-nest, nest-type-value,
+                        sub-message ]
               doc:
                 description: Documentation of the attribute.
                 type: string
@@ -261,6 +262,17 @@ properties:
                 description: Name of the struct type used for the attribute.
                 type: string
               # End genetlink-legacy
+              # Start netlink-raw
+              sub-message:
+                description:
+                  Name of the sub-message definition to use for the attribute.
+                type: string
+              selector:
+                description:
+                  Name of the attribute to use for dynamic selection of sub-message
+                  format specifier.
+                type: string
+              # End netlink-raw
 
       # Make sure name-prefix does not appear in subsets (subsets inherit naming)
       dependencies:
@@ -283,6 +295,43 @@ properties:
             items:
               required: [ type ]
 
+  # Start netlink-raw
+  sub-messages:
+    description: Definition of sub message attributes
+    type: array
+    items:
+      type: object
+      additionalProperties: False
+      required: [ name, formats ]
+      properties:
+        name:
+          description: Name of the sub-message definition
+          type: string
+        formats:
+          description: Dynamically selected format specifiers
+          type: array
+          items:
+            type: object
+            additionalProperties: False
+            required: [ value ]
+            properties:
+              value:
+                description:
+                  Value to match for dynamic selection of sub-message format
+                  specifier.
+                type: string
+              fixed-header:
+                description:
+                  Name of the struct definition to use as the fixed header
+                  for the sub message.
+                type: string
+              attribute-set:
+                description:
+                  Name of the attribute space from which to resolve attributes
+                  in the sub message.
+                type: string
+  # End netlink-raw
+
   operations:
     description: Operations supported by the protocol.
     type: object
-- 
2.42.0

Re: [PATCH net-next v1 2/6] doc/netlink: Add sub-message support to netlink-raw

[Jakub Kicinski]

On Thu, 30 Nov 2023 21:49:54 +0000 Donald Hunter wrote:
> Add a 'sub-message' attribute type with a selector that supports
> polymorphic attribute formats for raw netlink families like tc.
> 
> A sub-message attribute uses the value of another attribute as a
> selector key to choose the right sub-message format. For example if the
> following attribute has already been decoded:
> 
>   { "kind": "gre" }
> 
> and we encounter the following attribute spec:
> 
>   -
>     name: data
>     type: sub-message
>     sub-message: linkinfo-data-msg
>     selector: kind
> 
> Then we look for a sub-message definition called 'linkinfo-data-msg' and
> use the value of the 'kind' attribute i.e. 'gre' as the key to choose
> the correct format for the sub-message:
> 
>   sub-messages:
>     name: linkinfo-data-msg
>     formats:
>       -
>         value: bridge
>         attribute-set: linkinfo-bridge-attrs
>       -
>         value: gre
>         attribute-set: linkinfo-gre-attrs
>       -
>         value: geneve
>         attribute-set: linkinfo-geneve-attrs
> 
> This would decode the attribute value as a sub-message with the
> attribute-set called 'linkinfo-gre-attrs' as the attribute space.
> 
> A sub-message can have an optional 'fixed-header' followed by zero or
> more attributes from an attribute-set. For example the following
> 'tc-options-msg' sub-message defines message formats that use a mixture
> of fixed-header, attribute-set or both together:
> 
>   sub-messages:
>     -
>       name: tc-options-msg
>       formats:
>         -
>           value: bfifo
>           fixed-header: tc-fifo-qopt
>         -
>           value: cake
>           attribute-set: tc-cake-attrs
>         -
>           value: netem
>           fixed-header: tc-netem-qopt
>           attribute-set: tc-netem-attrs

SGTM, could you add all the info from the commit message somewhere 
in the documentation? Perhaps a new section at the end of
Documentation/userspace-api/netlink/specs.rst

> @@ -261,6 +262,17 @@ properties:
>                  description: Name of the struct type used for the attribute.
>                  type: string
>                # End genetlink-legacy
> +              # Start netlink-raw
> +              sub-message:
> +                description:
> +                  Name of the sub-message definition to use for the attribute.
> +                type: string
> +              selector:
> +                description:
> +                  Name of the attribute to use for dynamic selection of sub-message
> +                  format specifier.
> +                type: string

We can leave it for later either way, but have you seen any selectors
which would key on an integer, rather than a string?

Re: [PATCH net-next v1 2/6] doc/netlink: Add sub-message support to netlink-raw

[Donald Hunter]

Jakub Kicinski <kuba@kernel.org> writes:
>
> SGTM, could you add all the info from the commit message somewhere 
> in the documentation? Perhaps a new section at the end of
> Documentation/userspace-api/netlink/specs.rst

Ack, will do.

>> @@ -261,6 +262,17 @@ properties:
>>                  description: Name of the struct type used for the attribute.
>>                  type: string
>>                # End genetlink-legacy
>> +              # Start netlink-raw
>> +              sub-message:
>> +                description:
>> +                  Name of the sub-message definition to use for the attribute.
>> +                type: string
>> +              selector:
>> +                description:
>> +                  Name of the attribute to use for dynamic selection of sub-message
>> +                  format specifier.
>> +                type: string
>
> We can leave it for later either way, but have you seen any selectors
> which would key on an integer, rather than a string?

From what I have seen, I am fairly sure it's always going to be a
string. (famous last words)