Re: [RFC net-next 0/4] ynl: YAML netlink protocol descriptions

[Stephen Hemminger <stephen@networkplumber.org>]

On Wed, 10 Aug 2022 19:23:00 -0700
Jakub Kicinski <kuba@kernel.org> wrote:

> Netlink seems simple and reasonable to those who understand it.
> It appears cumbersome and arcane to those who don't.
> 
> This RFC introduces machine readable netlink protocol descriptions
> in YAML, in an attempt to make creation of truly generic netlink
> libraries a possibility. Truly generic netlink library here means
> a library which does not require changes to support a new family
> or a new operation.
> 
> Each YAML spec lists attributes and operations the family supports.
> The specs are fully standalone, meaning that there is no dependency
> on existing uAPI headers in C. Numeric values of all attribute types,
> operations, enums, and defines and listed in the spec (or unambiguous).
> This property removes the need to manually translate the headers for
> languages which are not compatible with C.
> 
> The expectation is that the spec can be used to either dynamically
> translate between whatever types the high level language likes (see
> the Python example below) or codegen a complete libarary / bindings
> for a netlink family at compilation time (like popular RPC libraries
> do).
> 
> Currently only genetlink is supported, but the "old netlink" should
> be supportable as well (I don't need it myself).
> 
> On the kernel side the YAML spec can be used to generate:
>  - the C uAPI header
>  - documentation of the protocol as a ReST file
>  - policy tables for input attribute validation
>  - operation tables
> 
> We can also codegen parsers and dump helpers, but right now the level
> of "creativity & cleverness" when it comes to netlink parsing is so
> high it's quite hard to generalize it for most families without major
> refactoring.
> 
> Being able to generate the header, documentation and policy tables
> should balance out the extra effort of writing the YAML spec.
> 
> Here is a Python example I promised earlier:
> 
>   ynl = YnlFamily("path/to/ethtool.yaml")
>   channels = ynl.channels_get({'header': {'dev_name': 'eni1np1'}})
> 
> If the call was successful "channels" will hold a standard Python dict,
> e.g.:
> 
>   {'header': {'dev_index': 6, 'dev_name': 'eni1np1'},
>    'combined_max': 1,
>    'combined_count': 1}
> 
> for a netdevsim device with a single combined queue.
> 
> YnlFamily is an implementation of a YAML <> netlink translator (patch 3).
> It takes a path to the YAML spec - hopefully one day we will make
> the YAMLs themselves uAPI and distribute them like we distribute
> C headers. Or get them distributed to a standard search path another
> way. Until then, the YNL library needs a full path to the YAML spec and
> application has to worry about the distribution of those.
> 
> The YnlFamily reads all the info it needs from the spec, resolves
> the genetlink family id, and creates methods based on the spec.
> channels_get is such a dynamically-generated method (i.e. grep for
> channels_get in the python code shows nothing). The method can be called
> passing a standard Python dict as an argument. YNL will look up each key
> in the YAML spec and render the appropriate binary (netlink TLV)
> representation of the value. It then talks thru a netlink socket
> to the kernel, and deserilizes the response, converting the netlink
> TLVs into Python types and constructing a dictionary.
> 
> Again, the YNL code is completely generic and has no knowledge specific
> to ethtool. It's fairly simple an incomplete (in terms of types
> for example), I wrote it this afternoon. I'm also pretty bad at Python,
> but it's the only language I can type which allows the method
> magic, so please don't judge :) I have a rather more complete codegen
> for C, with support for notifications, kernel -> user policy/type
> verification, resolving extack attr offsets into a path
> of attribute names etc, etc. But that stuff needs polishing and
> is less suitable for an RFC.
> 
> The ability for a high level language like Python to talk to the kernel
> so easily, without ctypes, manually packing structs, copy'n'pasting
> values for defines etc. excites me more than C codegen, anyway.
> 
> 
> Patch 1 adds a bit of documentation under Documentation/, it talks
> more about the schemas themselves.
> 
> Patch 2 contains the YAML schema for the YAML specs.
> 
> Patch 3 adds the YNL Python library.
> 
> Patch 4 adds a sample schema for ethtool channels and a demo script.
> 
> 
> Jakub Kicinski (4):
>   ynl: add intro docs for the concept
>   ynl: add the schema for the schemas
>   ynl: add a sample python library
>   ynl: add a sample user for ethtool
> 
>  Documentation/index.rst                     |   1 +
>  Documentation/netlink/bindings/ethtool.yaml | 115 +++++++
>  Documentation/netlink/index.rst             |  13 +
>  Documentation/netlink/netlink-bindings.rst  | 104 ++++++
>  Documentation/netlink/schema.yaml           | 242 ++++++++++++++
>  tools/net/ynl/samples/ethtool.py            |  30 ++
>  tools/net/ynl/samples/ynl.py                | 342 ++++++++++++++++++++
>  7 files changed, 847 insertions(+)
>  create mode 100644 Documentation/netlink/bindings/ethtool.yaml
>  create mode 100644 Documentation/netlink/index.rst
>  create mode 100644 Documentation/netlink/netlink-bindings.rst
>  create mode 100644 Documentation/netlink/schema.yaml
>  create mode 100755 tools/net/ynl/samples/ethtool.py
>  create mode 100644 tools/net/ynl/samples/ynl.py
> 

Would rather this be part of iproute2 rather than requiring it
to be maintained separately and part of the kernel tree.