[RFC net-next 0/4] tools/net/ynl: Add batch operations for nftables

[RFC net-next 0/4] tools/net/ynl: Add batch operations for nftables

[Donald Hunter]

The nftables netlink families use batch operations for create update and
delete operations. This is a first cut at extending the netlink-raw
schema so that operations can wrapped with begin-batch and end-batch
messages.

The begin/end messages themselves are defined as ordinary ops, but there
are new attributes that describe the op name and parameters for the
begin/end messages.

The section of yaml spec that defines the begin/end ops looks like this;
the newtable op is marked 'is-batch: true' so the message needs to be
wrapped with 'batch-begin(res-id: 10)' and batch-end(res-id: 10) messages:

operations:
  enum-model: directional
  begin-batch:                # Define how to begin a batch
    operation: batch-begin
    parameters:
      res-id: 10
  end-batch:                  # Define how to end a batch
    operation: batch-end
    parameters:
      res-id: 10
  list:
    -
      name: batch-begin
      doc: Start a batch of operations
      attribute-set: batch-attrs
      fixed-header: nfgenmsg
      do:
        request:
          value: 0x10
          attributes:
            - genid
        reply:
          value: 0x10
          attributes:
            - genid
    -
      name: batch-end
      doc: Finish a batch of operations
      attribute-set: batch-attrs
      fixed-header: nfgenmsg
      do:
        request:
          value: 0x11
          attributes:
            - genid
    -
      name: newtable
      doc: Create a new table.
      attribute-set: table-attrs
      fixed-header: nfgenmsg
      do:
        request:
          value: 0xa00
          is-batch: True      # This message must be in a batch
          attributes:
            - name

The code in ynl.py is sufficient to test the idea but I haven't extended
nlspec.py nor have I added any support for multiple messages to ynl.

This can be tested with e.g.:

./tools/net/ynl/cli.py --spec Documentation/netlink/specs/nftables.yaml \
     --do newtable --json '{"name": "table", "nfgen-family": 1}'

If the approach is acceptable, then I would do the following:

 - Extend nlspec.py to support the new schema properties.
 - Extend cli.py to include a --batch option, then only allow
   'is-batch' ops. Also fail 'is-batch' ops when --batch is not used.
 - Extend ynl to support a heterogeneous list of ops to be sent
   in a batch.
 - Update documentation.

I'm thinking that usage would be '--do <op> | --dump <op> | --batch' and
when '--batch' is used, the '--json' parameter would be a list of op /
param pairs like this:

[ { "newtable": { "name": "x", "nfgen-family": 1 },
  { "newchain": { "table": "x", "name": "y", "nfgen-family": 1 } ]

Alternatively, usage could be '--batch <ops>' where <ops> is the json
above.

Thoughts?

Donald Hunter (4):
  doc/netlink: Add batch op definitions to netlink-raw schema
  tools/net/ynl: Extract message encoding into _encode_message()
  tools/net/ynl: Add batch message encoding for nftables
  doc/netlink/specs: Add draft nftables spec

 Documentation/netlink/netlink-raw.yaml    |   21 +
 Documentation/netlink/specs/nftables.yaml | 1292 +++++++++++++++++++++
 tools/net/ynl/lib/ynl.py                  |   33 +-
 3 files changed, 1339 insertions(+), 7 deletions(-)
 create mode 100644 Documentation/netlink/specs/nftables.yaml

-- 
2.42.0

[RFC net-next 1/4] doc/netlink: Add batch op definitions to netlink-raw schema

[Donald Hunter]

The nftables netlink families use batch operations for create update and
delete operations. Extend the netlink-raw schema so that operations can
be marked as batch ops. Add definitions of the begin-batch and end-batch
messages.

The begin/end messages themselves are defined as ordinary ops, but there
are new attributes that describe the op name and parameters for the
begin/end messages.

The section of yaml spec that defines the begin/end ops looks like this;
the newtable op is marked 'is-batch: true' so the message needs to be
wrapped with 'batch-begin(res-id: 10)' and batch-end(res-id: 10) messages:

operations:
  enum-model: directional
  begin-batch:
    operation: batch-begin
    parameters:
      res-id: 10
  end-batch:
    operation: batch-end
    parameters:
      res-id: 10
  list:
    -
      name: batch-begin
      doc: Start a batch of operations
      attribute-set: batch-attrs
      fixed-header: nfgenmsg
      do:
        request:
          value: 0x10
          attributes:
            - genid
        reply:
          value: 0x10
          attributes:
            - genid
    -
      name: batch-end
      doc: Finish a batch of operations
      attribute-set: batch-attrs
      fixed-header: nfgenmsg
      do:
        request:
          value: 0x11
          attributes:
            - genid
    -
      name: newtable
      doc: Create a new table.
      attribute-set: table-attrs
      fixed-header: nfgenmsg
      do:
        request:
          value: 0xa00
          is-batch: True
          attributes:
            - name

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/netlink/netlink-raw.yaml | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/Documentation/netlink/netlink-raw.yaml b/Documentation/netlink/netlink-raw.yaml
index ac4e05415f2f..eb35fee44898 100644
--- a/Documentation/netlink/netlink-raw.yaml
+++ b/Documentation/netlink/netlink-raw.yaml
@@ -446,6 +446,11 @@ properties:
                         i.e. requests and responses have different message enums.
                       $ref: '#/$defs/uint'
                     # End genetlink-legacy
+                    # Start netlink-raw
+                    is-batch:
+                      description: Must be part of a message batch
+                      type: boolean
+                    # End netlink-raw
                 reply: *subop-attr-list
                 pre:
                   description: Hook for a function to run before the main callback (pre_doit or start).
@@ -469,6 +474,22 @@ properties:
             mcgrp:
               description: Name of the multicast group generating given notification.
               type: string
+      # Start netlink-raw
+      begin-batch: &batch-params
+        description: Definition of a message call for a batch message
+        type: object
+        additionalProperties: False
+        properties:
+          operation:
+            description: Name of the operation
+            type: string
+          parameters:
+            description: Parameters for the operation
+            type: object
+            items:
+              type: object
+      end-batch: *batch-params
+      # End netlink-raw
   mcast-groups:
     description: List of multicast groups.
     type: object
-- 
2.42.0

Re: [RFC net-next 1/4] doc/netlink: Add batch op definitions to netlink-raw schema

[Jakub Kicinski]

On Sun, 25 Feb 2024 17:46:16 +0000 Donald Hunter wrote:
> The nftables netlink families use batch operations for create update and
> delete operations. Extend the netlink-raw schema so that operations can
> be marked as batch ops. Add definitions of the begin-batch and end-batch
> messages.
> 
> The begin/end messages themselves are defined as ordinary ops, but there
> are new attributes that describe the op name and parameters for the
> begin/end messages.
> 
> The section of yaml spec that defines the begin/end ops looks like this;
> the newtable op is marked 'is-batch: true' so the message needs to be
> wrapped with 'batch-begin(res-id: 10)' and batch-end(res-id: 10) messages:

I'm not familiar with nftables nl. Can you explain what the batch ops
are for and how they function?

Begin / end makes it sound like some form of a transaction, is it?

Re: [RFC net-next 1/4] doc/netlink: Add batch op definitions to netlink-raw schema

[Donald Hunter]

Jakub Kicinski <kuba@kernel.org> writes:

> On Sun, 25 Feb 2024 17:46:16 +0000 Donald Hunter wrote:
>> The nftables netlink families use batch operations for create update and
>> delete operations. Extend the netlink-raw schema so that operations can
>> be marked as batch ops. Add definitions of the begin-batch and end-batch
>> messages.
>> 
>> The begin/end messages themselves are defined as ordinary ops, but there
>> are new attributes that describe the op name and parameters for the
>> begin/end messages.
>> 
>> The section of yaml spec that defines the begin/end ops looks like this;
>> the newtable op is marked 'is-batch: true' so the message needs to be
>> wrapped with 'batch-begin(res-id: 10)' and batch-end(res-id: 10) messages:
>
> I'm not familiar with nftables nl. Can you explain what the batch ops
> are for and how they function?
>
> Begin / end makes it sound like some form of a transaction, is it?

Yes, it's handled as a transaction, containing multiple messages wrapped
in BATCH_BEGIN / BATCH_END in a single skb.

The transaction batching could be implemented without any schema changes
by just adding multi-message capability to ynl. Then it would be the
caller's responsibility to specify the right begin / end messages.

Re: [RFC net-next 1/4] doc/netlink: Add batch op definitions to netlink-raw schema

[Jakub Kicinski]

On Tue, 27 Feb 2024 16:52:40 +0000 Donald Hunter wrote:
> > I'm not familiar with nftables nl. Can you explain what the batch ops
> > are for and how they function?
> >
> > Begin / end makes it sound like some form of a transaction, is it?  
> 
> Yes, it's handled as a transaction, containing multiple messages wrapped
> in BATCH_BEGIN / BATCH_END in a single skb.
> 
> The transaction batching could be implemented without any schema changes
> by just adding multi-message capability to ynl. Then it would be the
> caller's responsibility to specify the right begin / end messages.

That's where I was going with my questions :)
Feels like we need to figure out a nice API at the library level
and/or CLI. That could be more generally useful if anyone wants
to save syscalls.

Re: [RFC net-next 1/4] doc/netlink: Add batch op definitions to netlink-raw schema

[Donald Hunter]

Jakub Kicinski <kuba@kernel.org> writes:

> On Tue, 27 Feb 2024 16:52:40 +0000 Donald Hunter wrote:
>> > I'm not familiar with nftables nl. Can you explain what the batch ops
>> > are for and how they function?
>> >
>> > Begin / end makes it sound like some form of a transaction, is it?  
>> 
>> Yes, it's handled as a transaction, containing multiple messages wrapped
>> in BATCH_BEGIN / BATCH_END in a single skb.
>> 
>> The transaction batching could be implemented without any schema changes
>> by just adding multi-message capability to ynl. Then it would be the
>> caller's responsibility to specify the right begin / end messages.
>
> That's where I was going with my questions :)
> Feels like we need to figure out a nice API at the library level
> and/or CLI. That could be more generally useful if anyone wants
> to save syscalls.

Yep, I'm probably guilty of trying to put too much into the schema
again.

From a library API perspective, it'll need to take a list of tuples,
e.g. something like this:

ynl.do_multi([ (method, vals, flags), ... ])

As for the CLI, it will likely take a bit of experimentation to find a
usable balance between args and json payload.

Re: [RFC net-next 1/4] doc/netlink: Add batch op definitions to netlink-raw schema

[Jakub Kicinski]

On Tue, 27 Feb 2024 17:36:29 +0000 Donald Hunter wrote:
> Yep, I'm probably guilty of trying to put too much into the schema
> again.
> 
> From a library API perspective, it'll need to take a list of tuples,
> e.g. something like this:
> 
> ynl.do_multi([ (method, vals, flags), ... ])

Either that or some form of "start batch", "exec batch".
And any do/dump request in between the two just gets recorded
and executed at the "exec" step.
Not sure what's cleaner from the user perspective.

> As for the CLI, it will likely take a bit of experimentation to find a
> usable balance between args and json payload.

Yeah :( We may need manual walking of the args given not all do's will
have an associated JSON input :(

[RFC net-next 2/4] tools/net/ynl: Extract message encoding into _encode_message()

[Donald Hunter]

Make the message encoding a separate method so that it can be reused for
encoding batch messages.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 tools/net/ynl/lib/ynl.py | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/tools/net/ynl/lib/ynl.py b/tools/net/ynl/lib/ynl.py
index ac55aa5a3083..c6fc9588c235 100644
--- a/tools/net/ynl/lib/ynl.py
+++ b/tools/net/ynl/lib/ynl.py
@@ -831,6 +831,16 @@ class YnlFamily(SpecFamily):
 
       return op['do']['request']['attributes'].copy()
 
+    def _encode_message(self, op, vals, nl_flags, req_seq):
+        msg = self.nlproto.message(nl_flags, op.req_value, 1, req_seq)
+        if op.fixed_header:
+            msg += self._encode_struct(op.fixed_header, vals)
+        search_attrs = SpaceAttrs(op.attr_set, vals)
+        for name, value in vals.items():
+            msg += self._add_attr(op.attr_set.name, name, value, search_attrs)
+        msg = _genl_msg_finalize(msg)
+        return msg
+
     def _op(self, method, vals, flags=None, dump=False):
         op = self.ops[method]
 
@@ -841,13 +851,7 @@ class YnlFamily(SpecFamily):
             nl_flags |= Netlink.NLM_F_DUMP
 
         req_seq = random.randint(1024, 65535)
-        msg = self.nlproto.message(nl_flags, op.req_value, 1, req_seq)
-        if op.fixed_header:
-            msg += self._encode_struct(op.fixed_header, vals)
-        search_attrs = SpaceAttrs(op.attr_set, vals)
-        for name, value in vals.items():
-            msg += self._add_attr(op.attr_set.name, name, value, search_attrs)
-        msg = _genl_msg_finalize(msg)
+        msg = self._encode_message(op, vals, nl_flags, req_seq)
 
         self.sock.send(msg, 0)
 
-- 
2.42.0

[RFC net-next 3/4] tools/net/ynl: Add batch message encoding for nftables

[Donald Hunter]

The nftables families use batch operations for create, update and delete
operations. For ops that have 'is-batch: true' wrap them in begin-batch
and end-batch messages.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 tools/net/ynl/lib/ynl.py | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/tools/net/ynl/lib/ynl.py b/tools/net/ynl/lib/ynl.py
index c6fc9588c235..3a4af3c5a6a7 100644
--- a/tools/net/ynl/lib/ynl.py
+++ b/tools/net/ynl/lib/ynl.py
@@ -841,6 +841,12 @@ class YnlFamily(SpecFamily):
         msg = _genl_msg_finalize(msg)
         return msg
 
+    def _encode_batch_message(self, name, nl_flags, req_seq):
+        msg = self.yaml.get('operations').get(name)
+        op = self.ops[msg['operation']]
+        params = msg['parameters']
+        return self._encode_message(op, params, nl_flags, req_seq)
+
     def _op(self, method, vals, flags=None, dump=False):
         op = self.ops[method]
 
@@ -851,7 +857,16 @@ class YnlFamily(SpecFamily):
             nl_flags |= Netlink.NLM_F_DUMP
 
         req_seq = random.randint(1024, 65535)
-        msg = self._encode_message(op, vals, nl_flags, req_seq)
+        msg = b''
+
+        is_batch = op['do']['request'].get('is-batch', False)
+        if is_batch:
+            msg += self._encode_batch_message('begin-batch', nl_flags, req_seq)
+
+        msg += self._encode_message(op, vals, nl_flags, req_seq)
+
+        if is_batch:
+            msg += self._encode_batch_message('end-batch', nl_flags, req_seq)
 
         self.sock.send(msg, 0)
 
-- 
2.42.0

[RFC net-next 4/4] doc/netlink/specs: Add draft nftables spec

[Donald Hunter]

Add a spec for nftables that has nearly complete coverage of the ops,
but limited coverage of rule types and subexpressions.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/netlink/specs/nftables.yaml | 1292 +++++++++++++++++++++
 1 file changed, 1292 insertions(+)
 create mode 100644 Documentation/netlink/specs/nftables.yaml

diff --git a/Documentation/netlink/specs/nftables.yaml b/Documentation/netlink/specs/nftables.yaml
new file mode 100644
index 000000000000..74157f296f71
--- /dev/null
+++ b/Documentation/netlink/specs/nftables.yaml
@@ -0,0 +1,1292 @@
+# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)
+
+name: nftables
+protocol: netlink-raw
+protonum: 12
+
+doc:
+  Netfilter nftables configuration over netlink.
+
+definitions:
+  -
+    name: nfgenmsg
+    type: struct
+    members:
+      -
+        name: nfgen-family
+        type: u8
+      -
+        name: version
+        type: u8
+      -
+        name: res-id
+        byte-order: big-endian
+        type: u16
+  -
+    name: meta-keys
+    type: enum
+    entries:
+      - len
+      - protocol
+      - priority
+      - mark
+      - iif
+      - oif
+      - iifname
+      - oifname
+      - iftype
+      - oiftype
+      - skuid
+      - skgid
+      - nftrace
+      - rtclassid
+      - secmark
+      - nfproto
+      - l4-proto
+      - bri-iifname
+      - bri-oifname
+      - pkttype
+      - cpu
+      - iifgroup
+      - oifgroup
+      - cgroup
+      - prandom
+      - secpath
+      - iifkind
+      - oifkind
+      - bri-iifpvid
+      - bri-iifvproto
+      - time-ns
+      - time-day
+      - time-hour
+      - sdif
+      - sdifname
+      - bri-broute
+  -
+    name: cmp-ops
+    type: enum
+    entries:
+      - eq
+      - neq
+      - lt
+      - lte
+      - gt
+      - gte
+  -
+    name: object-type
+    type: enum
+    entries:
+      - unspec
+      - counter
+      - quota
+      - ct-helper
+      - limit
+      - connlimit
+      - tunnel
+      - ct-timeout
+      - secmark
+      - ct-expect
+      - synproxy
+  -
+    name: nat-range-flags
+    type: flags
+    entries:
+      - map-ips
+      - proto-specified
+      - proto-random
+      - persistent
+      - proto-random-fully
+      - proto-offset
+      - netmap
+  -
+    name: table-flags
+    type: flags
+    entries:
+      - dormant
+      - owner
+  -
+    name: chain-flags
+    type: flags
+    entries:
+      - base
+      - hw-offload
+      - binding
+  -
+    name: set-flags
+    type: flags
+    entries:
+      - anonymous
+      - constant
+      - interval
+      - map
+      - timeout
+      - eval
+      - object
+      - concat
+      - expr
+
+attribute-sets:
+  -
+    name: empty-attrs
+    attributes:
+      -
+        name: name
+        type: string
+  -
+    name: batch-attrs
+    attributes:
+      -
+        name: genid
+        type: u32
+        byte-order: big-endian
+  -
+    name: table-attrs
+    attributes:
+      -
+        name: name
+        type: string
+        doc: name of the table
+      -
+        name: flags
+        type: u32
+        byte-order: big-endian
+        doc: bitmask of flags
+        enum: table-flags
+        enum-as-flags: true
+      -
+        name: use
+        type: u32
+        byte-order: big-endian
+        doc: number of chains in this table
+      -
+        name: handle
+        type: u64
+        byte-order: big-endian
+        doc: numeric handle of the table
+      -
+        name: userdata
+        type: binary
+        doc: user data
+  -
+    name: chain-attrs
+    attributes:
+      -
+        name: table
+        type: string
+        doc: name of the table containing the chain
+      -
+        name: handle
+        type: u64
+        byte-order: big-endian
+        doc: numeric handle of the chain
+      -
+        name: name
+        type: string
+        doc: name of the chain
+      -
+        name: hook
+        type: nest
+        nested-attributes: nft-hook-attrs
+        doc: hook specification for basechains
+      -
+        name: policy
+        type: u32
+        byte-order: big-endian
+        doc: numeric policy of the chain
+      -
+        name: use
+        type: u32
+        byte-order: big-endian
+        doc: number of references to this chain
+      -
+        name: type
+        type: string
+        doc: type name of the chain
+      -
+        name: counters
+        type: nest
+        nested-attributes: nft-counter-attrs
+        doc: counter specification of the chain
+      -
+        name: flags
+        type: u32
+        byte-order: big-endian
+        doc: chain flags
+        enum: chain-flags
+        enum-as-flags: true
+      -
+        name: id
+        type: u32
+        byte-order: big-endian
+        doc: uniquely identifies a chain in a transaction
+      -
+        name: userdata
+        type: binary
+        doc: user data
+  -
+    name: counter-attrs
+    attributes:
+      -
+        name: bytes
+        type: u64
+        byte-order: big-endian
+      -
+        name: packets
+        type: u64
+        byte-order: big-endian
+      -
+        name: pad
+        type: pad
+  -
+    name: nft-hook-attrs
+    attributes:
+      -
+        name: num
+        type: u32
+        byte-order: big-endian
+      -
+        name: priority
+        type: s32
+        byte-order: big-endian
+      -
+        name: dev
+        type: string
+        doc: net device name
+      -
+        name: devs
+        type: nest
+        nested-attributes: hook-dev-attrs
+        doc: list of net devices
+  -
+    name: hook-dev-attrs
+    attributes:
+      -
+        name: name
+        type: string
+        multi-attr: true
+  -
+    name: nft-counter-attrs
+    attributes:
+      -
+        name: bytes
+        type: u64
+      -
+        name: packets
+        type: u64
+  -
+    name: rule-attrs
+    attributes:
+      -
+        name: table
+        type: string
+        doc: name of the table containing the rule
+      -
+        name: chain
+        type: string
+        doc: name of the chain containing the rule
+      -
+        name: handle
+        type: u64
+        byte-order: big-endian
+        doc: numeric handle of the rule
+      -
+        name: expressions
+        type: nest
+        nested-attributes: expr-list-attrs
+        doc: list of expressions
+      -
+        name: compat
+        type: nest
+        nested-attributes: rule-compat-attrs
+        doc: compatibility specifications of the rule
+      -
+        name: position
+        type: u64
+        byte-order: big-endian
+        doc: numeric handle of the previous rule
+      -
+        name: userdata
+        type: binary
+        doc: user data
+      -
+        name: id
+        type: u32
+        doc: uniquely identifies a rule in a transaction
+      -
+        name: position-id
+        type: u32
+        doc: transaction unique identifier of the previous rule
+      -
+        name: chain-id
+        type: u32
+        doc: add the rule to chain by ID, alternative to chain name
+  -
+    name: expr-list-attrs
+    attributes:
+      -
+        name: elem
+        type: nest
+        nested-attributes: expr-attrs
+        multi-attr: true
+  -
+    name: expr-attrs
+    attributes:
+      -
+        name: name
+        type: string
+        doc: name of the expression type
+      -
+        name: data
+        type: sub-message
+        sub-message: expr-ops
+        selector: name
+        doc: type specific data
+  -
+    name: rule-compat-attrs
+    attributes:
+      -
+        name: proto
+        type: binary
+        doc: numeric value of the handled protocol
+      -
+        name: flags
+        type: binary
+        doc: bitmask of flags
+  -
+    name: set-attrs
+    attributes:
+      -
+        name: table
+        type: string
+        doc: table name
+      -
+        name: name
+        type: string
+        doc: set name
+      -
+        name: flags
+        type: u32
+        enum: set-flags
+        byte-order: big-endian
+        doc: bitmask of enum nft_set_flags
+      -
+        name: key-type
+        type: u32
+        byte-order: big-endian
+        doc: key data type, informational purpose only
+      -
+        name: key-len
+        type: u32
+        byte-order: big-endian
+        doc: key data length
+      -
+        name: data-type
+        type: u32
+        byte-order: big-endian
+        doc: mapping data type
+      -
+        name: data-len
+        type: u32
+        byte-order: big-endian
+        doc: mapping data length
+      -
+        name: policy
+        type: u32
+        byte-order: big-endian
+        doc: selection policy
+      -
+        name: desc
+        type: nest
+        nested-attributes: set-desc-attrs
+        doc: set description
+      -
+        name: id
+        type: u32
+        doc: uniquely identifies a set in a transaction
+      -
+        name: timeout
+        type: u64
+        doc: default timeout value
+      -
+        name: gc-interval
+        type: u32
+        doc: garbage collection interval
+      -
+        name: userdata
+        type: binary
+        doc: user data
+      -
+        name: pad
+        type: pad
+      -
+        name: obj-type
+        type: u32
+        byte-order: big-endian
+        doc: stateful object type
+      -
+        name: handle
+        type: u64
+        byte-order: big-endian
+        doc: set handle
+      -
+        name: expr
+        type: nest
+        nested-attributes: expr-attrs
+        doc: set expression
+        multi-attr: true
+      -
+        name: expressions
+        type: nest
+        nested-attributes: set-list-attrs
+        doc: list of expressions
+  -
+    name: set-desc-attrs
+    attributes:
+      -
+        name: size
+        type: u32
+        byte-order: big-endian
+        doc: number of elements in set
+      -
+        name: concat
+        type: nest
+        nested-attributes: set-desc-concat-attrs
+        doc: description of field concatenation
+        multi-attr: true
+  -
+    name: set-desc-concat-attrs
+    attributes:
+      -
+        name: elem
+        type: nest
+        nested-attributes: set-field-attrs
+  -
+    name: set-field-attrs
+    attributes:
+      -
+        name: len
+        type: u32
+        byte-order: big-endian
+  -
+    name: set-list-attrs
+    attributes:
+      -
+        name: elem
+        type: nest
+        nested-attributes: expr-attrs
+        multi-attr: true
+  -
+    name: setelem-attrs
+    attributes:
+      -
+        name: key
+        type: nest
+        nested-attributes: data-attrs
+        doc: key value
+      -
+        name: data
+        type: nest
+        nested-attributes: data-attrs
+        doc: data value of mapping
+      -
+        name: flags
+        type: binary
+        doc: bitmask of nft_set_elem_flags
+      -
+        name: timeout
+        type: u64
+        doc: timeout value
+      -
+        name: expiration
+        type: u64
+        doc: expiration time
+      -
+        name: userdata
+        type: binary
+        doc: user data
+      -
+        name: expr
+        type: nest
+        nested-attributes: expr-attrs
+        doc: expression
+      -
+        name: objref
+        type: string
+        doc: stateful object reference
+      -
+        name: key-end
+        type: nest
+        nested-attributes: TODO
+        doc: closing key value
+      -
+        name: expressions
+        type: nest
+        nested-attributes: list-attrs
+        doc: list of expressions
+  -
+    name: setelem-list-elem-attrs
+    attributes:
+      -
+        name: elem
+        type: nest
+        nested-attributes: setelem-attrs
+        multi-attr: true
+  -
+    name: setelem-list-attrs
+    attributes:
+      -
+        name: table
+        type: string
+      -
+        name: set
+        type: string
+      -
+        name: elements
+        type: nest
+        nested-attributes: setelem-list-elem-attrs
+      -
+        name: set-id
+        type: u32
+  -
+    name: gen-attrs
+    attributes:
+      -
+        name: id
+        type: u32
+        byte-order: big-endian
+        doc: ruleset generation id
+      -
+        name: proc-pid
+        type: u32
+        byte-order: big-endian
+      -
+        name: proc-name
+        type: string
+  -
+    name: obj-attrs
+    attributes:
+      -
+        name: table
+        type: string
+        doc: name of the table containing the expression
+      -
+        name: name
+        type: string
+        doc: name of this expression type
+      -
+        name: type
+        type: u32
+        enum: object-type
+        byte-order: big-endian
+        doc: stateful object type
+      -
+        name: data
+        type: sub-message
+        sub-message: obj-data
+        selector: type
+        doc: stateful object data
+      -
+        name: use
+        type: u32
+        byte-order: big-endian
+        doc: number of references to this expression
+      -
+        name: handle
+        type: u64
+        byte-order: big-endian
+        doc: object handle
+      -
+        name: pad
+        type: pad
+      -
+        name: userdata
+        type: binary
+        doc: user data
+  -
+    name: quota-attrs
+    attributes:
+      -
+        name: bytes
+        type: u64
+        byte-order: big-endian
+      -
+        name: flags # TODO
+        type: u32
+        byte-order: big-endian
+      -
+        name: pad
+        type: pad
+      -
+        name: consumed
+        type: u64
+        byte-order: big-endian
+  -
+    name: flowtable-attrs
+    attributes:
+      -
+        name: table
+        type: string
+      -
+        name: name
+        type: string
+      -
+        name: hook
+        type: nest
+        nested-attributes: flowtable-hook-attrs
+      -
+        name: use
+        type: u32
+        byte-order: big-endian
+      -
+        name: handle
+        type: u64
+        byte-order: big-endian
+      -
+        name: pad
+        type: pad
+      -
+        name: flags
+        type: u32
+        byte-order: big-endian
+  -
+    name: flowtable-hook-attrs
+    attributes:
+      -
+        name: num
+        type: u32
+        byte-order: big-endian
+      -
+        name: priority
+        type: u32
+        byte-order: big-endian
+      -
+        name: devs
+        type: nest
+        nested-attributes: hook-dev-attrs
+  -
+    name: expr-cmp-attrs
+    attributes:
+      -
+        name: sreg
+        type: u32
+        byte-order: big-endian
+      -
+        name: op
+        type: u32
+        byte-order: big-endian
+        enum: cmp-ops
+      -
+        name: data
+        type: nest
+        nested-attributes: data-attrs
+  -
+    name: data-attrs
+    attributes:
+      -
+        name: value
+        type: binary
+        # sub-type: u8
+      -
+        name: verdict
+        type: nest
+        nested-attributes: verdict-attrs
+  -
+    name: verdict-attrs
+    attributes:
+      -
+        name: code
+        type: u32
+        byte-order: big-endian
+      -
+        name: chain
+        type: string
+      -
+        name: chain-id
+        type: u32
+  -
+    name: expr-counter-attrs
+    attributes:
+      -
+        name: bytes
+        type: u64
+        doc: Number of bytes
+      -
+        name: packets
+        type: u64
+        doc: Number of packets
+      -
+        name: pad
+        type: pad
+  -
+    name: expr-flow-offload-attrs
+    attributes:
+      -
+        name: name
+        type: string
+        doc: Flow offload table name
+  -
+    name: expr-immediate-attrs
+    attributes:
+      -
+        name: dreg
+        type: u32
+        byte-order: big-endian
+      -
+        name: data
+        type: nest
+        nested-attributes: data-attrs
+  -
+    name: expr-meta-attrs
+    attributes:
+      -
+        name: dreg
+        type: u32
+        byte-order: big-endian
+      -
+        name: key
+        type: u32
+        byte-order: big-endian
+        enum: meta-keys
+      -
+        name: sreg
+        type: u32
+        byte-order: big-endian
+  -
+    name: expr-nat-attrs
+    attributes:
+      -
+        name: type
+        type: u32
+        byte-order: big-endian
+      -
+        name: family
+        type: u32
+        byte-order: big-endian
+      -
+        name: reg-addr-min
+        type: u32
+        byte-order: big-endian
+      -
+        name: reg-addr-max
+        type: u32
+        byte-order: big-endian
+      -
+        name: reg-proto-min
+        type: u32
+        byte-order: big-endian
+      -
+        name: reg-proto-max
+        type: u32
+        byte-order: big-endian
+      -
+        name: flags
+        type: u32
+        byte-order: big-endian
+        enum: nat-range-flags
+        enum-as-flags: true
+  -
+    name: expr-payload-attrs
+    attributes:
+      -
+        name: dreg
+        type: u32
+        byte-order: big-endian
+      -
+        name: base
+        type: u32
+        byte-order: big-endian
+      -
+        name: offset
+        type: u32
+        byte-order: big-endian
+      -
+        name: len
+        type: u32
+        byte-order: big-endian
+      -
+        name: sreg
+        type: u32
+        byte-order: big-endian
+      -
+        name: csum-type
+        type: u32
+        byte-order: big-endian
+      -
+        name: csum-offset
+        type: u32
+        byte-order: big-endian
+      -
+        name: csum-flags
+        type: u32
+        byte-order: big-endian
+  -
+    name: expr-tproxy-attrs
+    attributes:
+      -
+        name: family
+        type: u32
+        byte-order: big-endian
+      -
+        name: reg-addr
+        type: u32
+        byte-order: big-endian
+      -
+        name: reg-port
+        type: u32
+        byte-order: big-endian
+
+sub-messages:
+  -
+    name: expr-ops
+    formats:
+      -
+        value: bitwise # TODO
+      -
+        value: cmp
+        attribute-set: expr-cmp-attrs
+      -
+        value: counter
+        attribute-set: expr-counter-attrs
+      -
+        value: ct # TODO
+      -
+        value: flow_offload
+        attribute-set: expr-flow-offload-attrs
+      -
+        value: immediate
+        attribute-set: expr-immediate-attrs
+      -
+        value: lookup # TODO
+      -
+        value: meta
+        attribute-set: expr-meta-attrs
+      -
+        value: nat
+        attribute-set: expr-nat-attrs
+      -
+        value: payload
+        attribute-set: expr-payload-attrs
+      -
+        value: tproxy
+        attribute-set: expr-tproxy-attrs
+  -
+    name: obj-data
+    formats:
+      -
+        value: counter
+        attribute-set: counter-attrs
+      -
+        value: quota
+        attribute-set: quota-attrs
+
+operations:
+  enum-model: directional
+  begin-batch:
+    operation: batch-begin
+    parameters:
+      res-id: 10
+  end-batch:
+    operation: batch-end
+    parameters:
+      res-id: 10
+  list:
+    -
+      name: batch-begin
+      doc: Start a batch of operations
+      attribute-set: batch-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0x10
+          attributes:
+            - genid
+        reply:
+          value: 0x10
+          attributes:
+            - genid
+    -
+      name: batch-end
+      doc: Finish a batch of operations
+      attribute-set: batch-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0x11
+          attributes:
+            - genid
+    -
+      name: newtable
+      doc: Create a new table.
+      attribute-set: table-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa00
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: gettable
+      doc: Get / dump tables.
+      attribute-set: table-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa01
+          attributes:
+            - name
+        reply:
+          value: 0xa00
+          attributes:
+            - name
+    -
+      name: deltable
+      doc: Delete an existing table.
+      attribute-set: table-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa02
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: destroytable
+      doc: Delete an existing table with destroy semantics (ignoring ENOENT errors).
+      attribute-set: table-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa1a
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: newchain
+      doc: Create a new chain.
+      attribute-set: chain-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa03
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: getchain
+      doc: Get / dump chains.
+      attribute-set: chain-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa04
+          attributes:
+            - name
+        reply:
+          value: 0xa03
+          attributes:
+            - name
+    -
+      name: delchain
+      doc: Delete an existing chain.
+      attribute-set: chain-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa05
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: destroychain
+      doc: Delete an existing chain with destroy semantics (ignoring ENOENT errors).
+      attribute-set: chain-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa1b
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: newrule
+      doc: Create a new rule.
+      attribute-set: rule-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa06
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: getrule
+      doc: Get / dump rules.
+      attribute-set: rule-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa07
+          attributes:
+            - name
+        reply:
+          value: 0xa06
+          attributes:
+            - name
+    -
+      name: getrule-reset
+      doc: Get / dump rules and reset stateful expressions.
+      attribute-set: rule-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa19
+          attributes:
+            - name
+        reply:
+          value: 0xa06
+          attributes:
+            - name
+    -
+      name: delrule
+      doc: Delete an existing rule.
+      attribute-set: rule-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa08
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: destroyrule
+      doc: Delete an existing rule with destroy semantics (ignoring ENOENT errors).
+      attribute-set: rule-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa1c
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: newset
+      doc: Create a new set.
+      attribute-set: set-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa09
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: getset
+      doc: Get / dump sets.
+      attribute-set: set-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa0a
+          attributes:
+            - name
+        reply:
+          value: 0xa09
+          attributes:
+            - name
+    -
+      name: delset
+      doc: Delete an existing set.
+      attribute-set: set-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa0b
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: destroyset
+      doc: Delete an existing set with destroy semantics (ignoring ENOENT errors).
+      attribute-set: set-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa1d
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: newsetelem
+      doc: Create a new set element.
+      attribute-set: setelem-list-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa0c
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: getsetelem
+      doc: Get / dump set elements.
+      attribute-set: setelem-list-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa0d
+          attributes:
+            - name
+        reply:
+          value: 0xa0c
+          attributes:
+            - name
+    -
+      name: getsetelem-reset
+      doc: Get / dump set elements and reset stateful expressions.
+      attribute-set: setelem-list-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa21
+          attributes:
+            - name
+        reply:
+          value: 0xa0c
+          attributes:
+            - name
+    -
+      name: delsetelem
+      doc: Delete an existing set element.
+      attribute-set: setelem-list-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa0e
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: destroysetelem
+      doc: Delete an existing set element with destroy semantics.
+      attribute-set: setelem-list-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa1e
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: getgen
+      doc: Get / dump rule-set generation.
+      attribute-set: gen-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa10
+          attributes:
+            - name
+        reply:
+          value: 0xa0f
+          attributes:
+            - name
+    -
+      name: newobj
+      doc: Create a new stateful object.
+      attribute-set: obj-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa12
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: getobj
+      doc: Get / dump stateful objects.
+      attribute-set: obj-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa13
+          attributes:
+            - name
+        reply:
+          value: 0xa12
+          attributes:
+            - name
+    -
+      name: delobj
+      doc: Delete an existing stateful object.
+      attribute-set: obj-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa14
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: destroyobj
+      doc: Delete an existing stateful object with destroy semantics.
+      attribute-set: obj-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa1f
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: newflowtable
+      doc: Create a new flow table.
+      attribute-set: flowtable-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa16
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: getflowtable
+      doc: Get / dump flow tables.
+      attribute-set: flowtable-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa17
+          attributes:
+            - name
+        reply:
+          value: 0xa16
+          attributes:
+            - name
+    -
+      name: delflowtable
+      doc: Delete an existing flow table.
+      attribute-set: flowtable-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa18
+          is-batch: True
+          attributes:
+            - name
+    -
+      name: destroyflowtable
+      doc: Delete an existing flow table with destroy semantics.
+      attribute-set: flowtable-attrs
+      fixed-header: nfgenmsg
+      do:
+        request:
+          value: 0xa20
+          is-batch: True
+          attributes:
+            - name
+
+mcast-groups:
+  list:
+    -
+      name: mgmt
-- 
2.42.0