[ANNOUNCE] nftables 1.1.2 release

[ANNOUNCE] nftables 1.1.2 release

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 6057 bytes --]

Hi!

The Netfilter project proudly presents:

        nftables 1.1.2

This release contains (in no particular order):

- Allow for protocol dependency on sets, eg.

    table inet test {
       set protos {
               typeof meta l4proto
               elements = { tcp, udp }
       }

       chain prerouting {
               type filter hook prerouting priority mangle; policy accept;
               meta l4proto @protos tproxy to :1088
       }
    }

- Support for more advanced bitwise operations with statements:

    ... ct mark set ct mark & 0xffff0000 | meta mark & 0xffff
    ... meta mark set meta mark & 0xffff0000 | meta cpu << 8 | iif

  This requires Linux kernel >= 6.13.

- Set element auto-merge now skips elements with timeout/expiration.

- Allow to use queue with typeof.

    table inet t {
       map get_queue_id {
               typeof ip saddr . ip daddr . tcp dport : queue
               elements = { 127.0.0.1 . 127.0.0.1 . 22 : 1,
                            127.0.0.1 . 127.0.0.2 . 22 : 2 }
       }

       chain test {
               queue flags bypass to ip saddr . ip daddr . tcp dport map @get_queue_id
       }
    }

- Memory footprint reduction for set elements.

- Update nft monitor to reports flowtable events.

- Allow for listing sets with:

    list sets inet foo

  for consistency with existing commands. Previous versions require the 'table'
  keyword for this to work, ie.

    list sets table inet foo

- Support for merging bitmask matching in set/map with -o/--optimize

     # nft -c -o -f ruleset.nft
     Merging:
     ruleset.nft:7:17-76:                 tcp flags & (fin | syn | rst | ack | urg) == fin | ack | urg
     ruleset.nft:8:17-70:                 tcp flags & (fin | syn | rst | ack | urg) == fin | ack
     ruleset.nft:9:17-64:                 tcp flags & (fin | syn | rst | ack | urg) == fin
     ruleset.nft:10:17-70:                 tcp flags & (fin | syn | rst | ack | urg) == syn | ack
     ruleset.nft:11:17-64:                 tcp flags & (fin | syn | rst | ack | urg) == syn
     ruleset.nft:12:17-70:                 tcp flags & (fin | syn | rst | ack | urg) == rst | ack
     ruleset.nft:13:17-64:                 tcp flags & (fin | syn | rst | ack | urg) == rst
     ruleset.nft:14:17-70:                 tcp flags & (fin | syn | rst | ack | urg) == ack | urg
     ruleset.nft:15:17-64:                 tcp flags & (fin | syn | rst | ack | urg) == ack
     into:
            tcp flags & (fin | syn | rst | ack | urg) == { fin | ack | urg, fin | ack, fin, syn | ack, syn, rst | ack, rst, ack | urg, ack }

- Use range expression to represent a range, instead of two comparisons.

    -  [ cmp gte reg 1 0x00005000 ]
    -  [ cmp lte reg 1 0x00005a00 ]
    +  [ range eq reg 1 0x00005000 0x00005a00 ]

- Improve mptcp support with symbol table for subtypes:

    set s13 {
             typeof tcp option mptcp subtype
             elements = { mp-join, dss }
    }

    # nft describe tcp option mptcp subtype
    exthdr expression, datatype integer (mptcp option subtype) (basetype integer), 4 bits

    pre-defined symbolic constants (in decimal):
        mp-capable                                         0
        mp-join                                            1
        dss                                                2
        add-addr                                           3
        remove-addr                                        4
        mp-prio                                            5
        mp-fail                                            6
        mp-fastclose                                       7
        mp-tcprst                                          8

- Support for mangling bitfield headers, eg.

    ... ip dscp set ip dscp | 0x1

- Print set element with multi-word description in single one line.
  If the set element:

    - represents a mapping
    - has a timeout
    - has a comment
    - has counter/quota/limit
    - concatenation (already printed in a single line before this patch)

  ie. if the set element requires several words, then print it in one
  single line, eg.

    table ip x {
          set y {
                typeof ip saddr
                counter
                elements = { 192.168.10.35 counter packets 0 bytes 0,
                             192.168.10.101 counter packets 0 bytes 0,
                             192.168.10.135 counter packets 0 bytes 0 }
          }
    }

- Fix extended error reporting with large set elements.

- Fix extended error reporting with large set elements.

- Fix incorrect removal of meta nfproto in listings.

      ... meta nfproto ipv4 ct mark 0x00000001
      ... meta nfproto ipv6 ct protocol 6

- Fix get command with interval sets/maps:

    # nft get element x y { 1.1.1.2 }
    table ip x {
            map y {
                    typeof ip saddr : meta mark
                    counter
                    flags interval,timeout
                    elements = { 1.1.1.1-1.1.1.10 timeout 10m : 20 }
            }
    }

- Fix reset command with interval sets/maps too:

    # nft reset element inet filter intervalset { 1.2.3.4 }

- Do not remove layer 4 protocol dependency when listing raw expressions, eg.

    meta l4proto 91 @th,0,16 0x0 accept

- Fixes for -o/--optimize.

- Support for typeof in JSON.

... and a bunch of assorted fixes, manpage updates too and tests enhancements.

See changelog for more details (attached to this email).

You can download this new release from:

https://www.netfilter.org/projects/nftables/downloads.html
https://www.netfilter.org/pub/nftables/

[ NOTE: We have switched to .tar.xz files for releases. ]

To build the code, libnftnl >= 1.2.9 and libmnl >= 1.0.4 are required:

* https://netfilter.org/projects/libnftnl/index.html
* https://netfilter.org/projects/libmnl/index.html

Visit our wikipage for user documentation at:

* https://wiki.nftables.org

For the manpage reference, check man(8) nft.

In case of bugs and feature requests, file them via:

* https://bugzilla.netfilter.org

Happy firewalling.

[-- Attachment #2: changes-nftables-1.1.2.txt --]
[-- Type: text/plain, Size: 8443 bytes --]

Donald Yandt (1):
      mnl: fix basehook comparison

Eric Long (1):
      libnftables-json: fix raw payload expression documentation

Florian Westphal (53):
      tests: shell: fix spurious dump failure in vmap timeout test
      tests: shell: don't rely on writable test directory
      tests: monitor: fix up test case breakage
      doc: extend description of fib expression
      src: allow to map key to nfqueue number
      tests: shell: add a test case for netdev ruleset flush + parallel link down
      tests: shell: add cgroupv2 socket match test case
      parser_bison: fix UaF when reporting table parse error
      rule: make cmd_free(NULL) valid
      evaluate: allow to re-use existing metered set
      netlink_delinarize: fix bogus munging of mask value
      src: add and use payload_expr_trim_force
      tests: py: extend raw payload match tests
      parser_bison: compact and simplify list and reset syntax
      parser_bison: get rid of unneeded statement
      payload: remove double-store
      payload: return early if dependency is not a payload expression
      tests: add atomic chain replace test
      tests: remove temporary file
      payload: don't kill dependency for proto_th
      tcpopt: add symbol table for mptcp suboptions
      expression: propagate key datatype for anonymous sets
      netlink_delinearize: also consider exthdr type when trimming binops
      expression: expr_build_udata_recurse should recurse
      segtree: fix string data initialisation
      doc: add mptcp to tcp option matching list
      src: fix reset element support for interval set type
      tests: extend reset test case to cover interval set and map type
      evaluate: don't crash if range has same start and end interval
      tests: shell: skip interval size tests on kernel that lack rbtree size fix
      evaluate: fix expression data corruption
      evaluate: don't allow merging interval set/map with non-interval one
      evaluate: move interval flag compat check after set key evaluation
      netlink: fix stack buffer overrun when emitting ranged expressions
      parser_bison: reject non-serializeable typeof expressions
      netlink_delinerize: add more restrictions on meta nfproto removal
      expression: tolerate named set protocol dependency
      evaluate: don't allow nat map with specified protocol
      rule: return error if table does not exist
      evaluate: fix assertion failure with malformed map definitions
      json: make sure timeout list is initialised
      evaluate: don't update cache for anonymous chains
      json: return error if table does not exist
      json: don't BUG when asked to list synproxies
      json: fix error propagation when parsing binop lhs/rhs
      expression: don't try to import empty string
      evaluate: compact STMT_F_STATEFUL checks
      evaluate: only allow stateful statements in set and map definitions
      cache: don't crash when filter is NULL
      evaluate: reject: remove unused expr function argument
      evaluate: fix crash when generating reject statement error
      parser_json: only allow concatenations with 2 or more expressions
      evaluate: bail out if ct saddr/daddr dependency cannot be inserted

Jan Engelhardt (1):
      build: add hint for a2x error message

Jeremy Sowden (1):
      src: allow binop expressions with variable right-hand operands

Pablo Neira Ayuso (73):
      src: collapse set element commands from parser
      mnl: rename to mnl_seqnum_alloc() to mnl_seqnum_inc()
      mnl: update cmd_add_loc() to take struct nlmsghdr
      rule: netlink attribute offset is uint32_t for struct nlerr_loc
      src: fix extended netlink error reporting with large set elements
      tests: shell: move device to different namespace
      json: collapse set element commands from parser
      datatype: remove unused flags field
      mnl: restore --debug=netlink output with chains
      optimize: compare expression length
      intervals: set internal element location with the deletion trigger
      expression: remove elem_flags from EXPR_SET_ELEM to shrink struct expr size
      src: remove unused token_offset from struct location
      src: remove last_line from struct location
      src: shrink line_offset in struct location to 4 bytes
      libnftables: include canonical path to avoid duplicates
      main: prepend error tag to printed errors when parsing options
      intervals: add helper function to set previous element
      intervals: do not merge intervals with different timeout
      src: add EXPR_RANGE_VALUE expression and use it
      rule: constify set_is_non_concat_range()
      mnl: rename list of expression in mnl_nft_setelem_batch()
      mnl: do not send set size when set is constant set
      src: rework singleton interval transformation to reduce memory consumption
      scanner: better error reporting for CRLF line terminators
      evaluate: remove variable shadowing
      tests: shell: use mount --bind to change cgroupsv2 root
      tests: shell: delete netdev chain after test
      exthdr: incomplete type 2 routing header definition
      datatype: clamp boolean value to 0 and 1
      ipopt: use ipv4 address datatype for address field in ip options
      parser_bison: turn redundant ip option type field match into boolean
      src: add symbol range expression to further compact intervals
      netlink_linearize: use range expression for OP_EQ and OP_IMPLICIT
      evaluate: auto-merge is only available for singleton interval sets
      tests: shell: interval sets with size
      tests: shell: random interval set with size
      evaluate: optimize zero length range
      evaluate: consolidate evaluation of symbol range expression
      payload: honor inner payload description in payload_expr_cmp()
      evaluate: release existing datatype when evaluating unary expression
      evaluate: simplify payload statement evaluation for bitfields
      evaluate: reject unsupported expressions in payload statement for bitfields
      evaluate: support for bitfield payload statement with binary operation
      netlink_delinearize: support for bitfield payload statement with binary operation
      segtree: incomplete output in get element command with maps
      netlink_linearize: reduce register waste with non-constant binop expressions
      src: print set element with multi-word description in single one line
      src: replace struct stmt_ops by type field in struct stmt
      tests: py: remove unknown fields
      parser_bison: consolidate counter grammar rule for set elements
      parser_bison: consolidate limit grammar rule for set elements
      parser_bison: consolidate quota grammar rule for set elements
      parser_bison: consolidate last grammar rule for set elements
      parser_bison: consolidate connlimit grammar rule for set elements
      tests: shell: extend coverage for set element statements
      tests: shell: missing ct count elements in new set_stmt test
      expression: add __EXPR_MAX and use it to define EXPR_MAX
      optimize: incorrect comparison for reject statement
      optimize: compact bitmask matching in set/map
      src: transform flag match expression to binop expression from parser
      src: remove flagcmp expression
      json: disallow empty concatenation
      expression: initialize list of expression to silence gcc compile warning
      expression: incorrect assert() list_expr_to_binop
      parser_json: reject empty jump/goto chain
      parser_json: allow statement stateful statement only in set elements
      parser_json: bail out on malformed statement in set
      cache: prevent possible crash rule filter is NULL
      optimize: expand expression list when merging into concatenation
      optimize: invalidate merge in case of duplicated key in set/map
      parser_bison: add selector_expr rule to restrict typeof_expr
      build: Bump version to 1.1.2

Phil Sutter (8):
      tests: shell: Join arithmetic statements in maps/vmap_timeout
      json: Support typeof in set and map types
      tests: py: Fix for storing payload into missing file
      monitor: Recognize flowtable add/del events
      tests: monitor: Run in own netns
      tests: monitor: Become $PWD agnostic
      tests: shell: Add socat availability feature test
      tests: shell: Fix owner/0002-persist on aarch64

Xiao Liang (1):
      fib: Change data type of fib oifname to "ifname"

Re: [ANNOUNCE] nftables 1.1.2 release

[Jan Engelhardt]

On Monday 2025-04-14 19:49, Pablo Neira Ayuso wrote:
>You can download this new release from:
>https://www.netfilter.org/pub/nftables/
>[ NOTE: We have switched to .tar.xz files for releases. ]

$ tar -tf nftables-1.1.2.tar.xz|grep main.nf
nftables-1.1.2/files/nftables/main.nft

This file I do not see it in the git repo.
main.nft is not autogenerated when running the usual
autoreconf/configure procedure from the git repo either.

main.nft was part of my earlier patch about adding a systemd unit,
but that was not applied yet either.

Re: [ANNOUNCE] nftables 1.1.2 release

[Pablo Neira Ayuso]

On Mon, Apr 14, 2025 at 10:19:48PM +0200, Jan Engelhardt wrote:
> 
> On Monday 2025-04-14 19:49, Pablo Neira Ayuso wrote:
> >You can download this new release from:
> >https://www.netfilter.org/pub/nftables/
> >[ NOTE: We have switched to .tar.xz files for releases. ]
> 
> $ tar -tf nftables-1.1.2.tar.xz|grep main.nf
> nftables-1.1.2/files/nftables/main.nft
> 
> This file I do not see it in the git repo.
> main.nft is not autogenerated when running the usual
> autoreconf/configure procedure from the git repo either.
> 
> main.nft was part of my earlier patch about adding a systemd unit,
> but that was not applied yet either.

That was my release script, it did not pick up on a clean clone,
it has picked up on tree including this leftover after local revert.

Re: [ANNOUNCE] nftables 1.1.2 release

[Slavko]

Dňa 14. 4. o 19:49 Pablo Neira Ayuso napísal(a):

> - Set element auto-merge now skips elements with timeout/expiration.

please, can you write more about this?

regards
-- 
Slavko
https://www.slavino.sk/

Re: [ANNOUNCE] nftables 1.1.2 release

[Pablo Neira Ayuso]

On Tue, Apr 15, 2025 at 09:58:52AM +0200, Slavko wrote:
> Dňa 14. 4. o 19:49 Pablo Neira Ayuso napísal(a):
> 
> > - Set element auto-merge now skips elements with timeout/expiration.
> 
> please, can you write more about this?

Sure:

https://git.netfilter.org/nftables/commit/?id=da0bac050c8b2588242727f9915a1ea8bc48ceb2

Re: [ANNOUNCE] nftables 1.1.2 release

[Slavko]

On 15. apríla 2025 14:39:20 UTC, Pablo Neira Ayuso <pablo@netfilter.org> wrote:

>https://git.netfilter.org/nftables/commit/?id=da0bac050c8b2588242727f9915a1ea8bc48ceb2

Thanks, but i still do not understand, consider this set:

    table inet filter {
        set testset {
                type ipv4_addr
                flags interval,timeout
                auto-merge
                timeout 1m
        }
    }

Now i add one network, and one or two seconds later second
network::

    nft add element inet filter testset "{ 192.168.1.0/24 }"
    sleep 1
    nft add element inet filter testset "{ 192.168.2.0/24 }"

Currently (1.0.6), they are merged with timeout reset::

    table inet filter {
        set testset {
                ...
                elements = { 192.168.1.0-192.168.2.255 expires XY }
        }
    }

What will be result with new behavior? Will be both (not merged)
in set, each with different timeout/expires?

Another example is to add subnet of existing element, currently
the new subnet is not added (or is merged into existing without
timeout change). How it will work with this new behavior? Will be
both in set? Or error happens? Or something other?

regards


-- 
Slavko
https://www.slavino.sk/

Re: [ANNOUNCE] nftables 1.1.2 release

[Pablo Neira Ayuso]

On Tue, Apr 15, 2025 at 03:22:52PM +0000, Slavko wrote:
> On 15. apríla 2025 14:39:20 UTC, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> 
> >https://git.netfilter.org/nftables/commit/?id=da0bac050c8b2588242727f9915a1ea8bc48ceb2
> 
> Thanks, but i still do not understand, consider this set:
> 
>     table inet filter {
>         set testset {
>                 type ipv4_addr
>                 flags interval,timeout
>                 auto-merge
>                 timeout 1m
>         }
>     }
> 
> Now i add one network, and one or two seconds later second
> network::
> 
>     nft add element inet filter testset "{ 192.168.1.0/24 }"
>     sleep 1
>     nft add element inet filter testset "{ 192.168.2.0/24 }"
> 
> Currently (1.0.6), they are merged with timeout reset::
> 
>     table inet filter {
>         set testset {
>                 ...
>                 elements = { 192.168.1.0-192.168.2.255 expires XY }
>         }
>     }
> 
> What will be result with new behavior? Will be both (not merged)
> in set, each with different timeout/expires?

After this update, two different intervals with different timeouts are
added.

> Another example is to add subnet of existing element, currently
> the new subnet is not added (or is merged into existing without
> timeout change). How it will work with this new behavior? Will be
> both in set? Or error happens? Or something other?

After this update, with subset, an error will be reported if the
interval overlaps.

Re: [ANNOUNCE] nftables 1.1.2 release

[Slavko]

On 15. apríla 2025 15:54:15 UTC, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>On Tue, Apr 15, 2025 at 03:22:52PM +0000, Slavko wrote:

>> Now i add one network, and one or two seconds later second
>> network::
>> 
>>     nft add element inet filter testset "{ 192.168.1.0/24 }"
>>     sleep 1
>>     nft add element inet filter testset "{ 192.168.2.0/24 }"
>> 

>After this update, two different intervals with different timeouts are
>added.

OK, that is good, and IMO expected.

>> Another example is to add subnet of existing element, currently
>> the new subnet is not added (or is merged into existing without
>> timeout change). How it will work with this new behavior? Will be
>> both in set? Or error happens? Or something other?
>
>After this update, with subset, an error will be reported if the
>interval overlaps.

That is not good, it will break my current use case -- set filled
from BGP, as from time to time networks of different ASNs
overlaps. In really, i use auto-merge in this set just due this...

I hope, that in one big atomic add, all timeouts will be the same
(set is flushed in this atomic step), but one cannot do it in cycle
(with separate add), as even ms are compared...

regards


-- 
Slavko
https://www.slavino.sk/

Re: [ANNOUNCE] nftables 1.1.2 release

[Pablo Neira Ayuso]

On Tue, Apr 15, 2025 at 04:19:43PM +0000, Slavko wrote:
> On 15. apríla 2025 15:54:15 UTC, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> >On Tue, Apr 15, 2025 at 03:22:52PM +0000, Slavko wrote:
> 
> >> Now i add one network, and one or two seconds later second
> >> network::
> >> 
> >>     nft add element inet filter testset "{ 192.168.1.0/24 }"
> >>     sleep 1
> >>     nft add element inet filter testset "{ 192.168.2.0/24 }"
> >> 
> 
> >After this update, two different intervals with different timeouts are
> >added.
> 
> OK, that is good, and IMO expected.
> 
> >> Another example is to add subnet of existing element, currently
> >> the new subnet is not added (or is merged into existing without
> >> timeout change). How it will work with this new behavior? Will be
> >> both in set? Or error happens? Or something other?
> >
> >After this update, with subset, an error will be reported if the
> >interval overlaps.
> 
> That is not good, it will break my current use case -- set filled
> from BGP, as from time to time networks of different ASNs
> overlaps. In really, i use auto-merge in this set just due this...
> 
> I hope, that in one big atomic add, all timeouts will be the same
> (set is flushed in this atomic step), but one cannot do it in cycle
> (with separate add), as even ms are compared...

Scenario 1) 192.168.2.0/24 exists
            192.168.2.10 is added with timeout X.

then, refresh 192.168.2.0/24 with new timeout X.

Scenario 2) 192.168.2.0/24 exists
            192.168.3.0/24 is added

then, refresh 192.168.2.0-192.168.3.255 with new timeout X.

Otherwise, auto-merge becomes of limited use with timers.

Let me spin over this again and get back to you, thanks for you
feedback.

Re: [ANNOUNCE] nftables 1.1.2 release

[Slavko]

On 15. apríla 2025 16:28:03 UTC, Pablo Neira Ayuso <pablo@netfilter.org> wrote:

>Otherwise, auto-merge becomes of limited use with timers.

I have timeout on it just for case, that something goes wrong
and content is not updated/flushed, to prevent stale elements
and auto-merge just due overlaping entries.

I use this approach with ipsets (i have only one public nft host
yet, for testing it). The main difference is, that i can have subnets
and its supernets (as separate entries) without any problems in
one set.

>Let me spin over this again and get back to you, thanks for you
>feedback.

You are welcome.

regards


-- 
Slavko
https://www.slavino.sk/