Issue with counter and interval map

Issue with counter and interval map

[Jann Haber]

[-- Attachment #1: Type: text/plain, Size: 1509 bytes --]

Dear nftables-developers,

at Selfnet, we have been operating our CGN based on nftables for roughly 4 years now (at that time we switched from iptables). Recently, we have upgraded our first server from Debian bullseye (Kernel 5.10, nftables 0.9.8) to bookworm (Kernel 6.1, nftables 1.0.6). On bookworm, our ruleset that works well on bullseye fails to load.

We have boiled it down to the minimal example attached, which fails to load correctly on bookworm and also on a current Arch-Linux.

xxxxx@xxxxx:~$ sudo nft -f example.conf
example.conf:5:35-48: Error: Could not process rule: No such file or directory
add element inet filter testmap { 192.168.0.0/24 : "TEST" }
                                   ^^^^^^^^^^^^^^
What we have tested:
- Removing the last line from the file and running it later manually via the command line, there is no error
- Splitting the file in two (having the final line in a separate file), the two files can be applied with two nft -f calls with no error
- When swapping the lines 3 and 4 (i.e. first add counter, then add map), there is no error applying the file
- Removing "flags: interval" from the map and testing with a single IP, there is no error applying the file

In summary, I believe our rule syntax is ok - but something is going wrong when the rules are applied in the given order atomically with "nft -f". We appreciate any insight, please also let us know if we did something wrong or if we can assist with debugging further.

Thank you and best Regards,
Jann

[-- Attachment #2: example.conf --]
[-- Type: text/plain, Size: 198 bytes --]

flush ruleset
add table inet filter
add map inet filter testmap { type ipv4_addr : counter; flags interval;}
add counter inet filter TEST
add element inet filter testmap { 192.168.0.0/24 : "TEST" }

Re: Issue with counter and interval map

[Pablo Neira Ayuso]

Hi Jann,

On Wed, Sep 13, 2023 at 07:42:01PM +0200, Jann Haber wrote:
> Dear nftables-developers,
> 
> at Selfnet, we have been operating our CGN based on nftables for
> roughly 4 years now (at that time we switched from iptables).
> Recently, we have upgraded our first server from Debian bullseye
> (Kernel 5.10, nftables 0.9.8) to bookworm (Kernel 6.1, nftables
> 1.0.6). On bookworm, our ruleset that works well on bullseye fails
> to load.
> 
> We have boiled it down to the minimal example attached, which fails
> to load correctly on bookworm and also on a current Arch-Linux.
> 
> xxxxx@xxxxx:~$ sudo nft -f example.conf
> example.conf:5:35-48: Error: Could not process rule: No such file or directory
> add element inet filter testmap { 192.168.0.0/24 : "TEST" }
>                                   ^^^^^^^^^^^^^^
> What we have tested:
> - Removing the last line from the file and running it later manually
>   via the command line, there is no error
> - Splitting the file in two (having the final line in a separate
>   file), the two files can be applied with two nft -f calls with no
>   error
> - When swapping the lines 3 and 4 (i.e. first add counter, then add
>   map), there is no error applying the file
> - Removing "flags: interval" from the map and testing with a single
>   IP, there is no error applying the file
> 
> In summary, I believe our rule syntax is ok - but something is going
> wrong when the rules are applied in the given order atomically with
> "nft -f". We appreciate any insight, please also let us know if we
> did something wrong or if we can assist with debugging further.

I can reproduce it, this is a userspace bug which happens with
interval sets in nft_cmd_post_expand(), I will post a fix asap.

> flush ruleset
> add table inet filter
> add map inet filter testmap { type ipv4_addr : counter; flags interval;}
> add counter inet filter TEST
> add element inet filter testmap { 192.168.0.0/24 : "TEST" }

Thanks for the detailed report, I will also add this test case to
improve coverage.

Re: Issue with counter and interval map

[Jann Haber]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

Hi Pablo,

thank you first of all for your prompt response, very appreciated!

We have a second issue also related to interval maps and counters, however this seems to be more along the lines of a userspace performance problem. At Selfnet, we will soon hit 7000 members, with our traffic running across two redundant CGN machines powered by nftables. Naturally, our maps and sets can become quite large and may have some tens of thousands of entries.

I have attached a small python script that creates an "example.conf" file containing a simple set of very repetitive rules including counters and 4 interval maps each of the size ~16000 entries referencing these counters. On our bookworm and ArchLinux hosts, the resulting rules take very long to load with "nft -f" (at least multiple minutes). All other maps in our ruleset, some of which have similar size, are loaded instantly, so the size itself seems not to be an issue for nft in general.
If needed I can also provide the example.conf file itself, but due to the size of a few MB, I rather avoid to spam the mailing list.

Further info:
- With a regular map instead of an interval map (just remove the "flags interval" in the example), the rules are loaded in fractions of a second
- Using add map { [...] elements = [...] } instead of add map { [...] }; add element [...]; add element [...] and so on, the ruleset is also loaded quickly (this is our workaround that we use for now)
- We have had no issues with this kind of ruleset on Debian Bullseye (Kernel 5.10, nftables 0.9.8), it seems to have been introduced later

Thanks again and looking forward to your feedback!

Kind Regards,
Jann

[-- Attachment #2: nftables-slow.py --]
[-- Type: text/x-python, Size: 1199 bytes --]



with open("example.conf", "w") as f:
    print("Initial lines")
    f.write("flush ruleset\n")
    f.write("add table inet filter\n")

    print("Counters")
    for i in range(32*256):
        f.write(f"add counter inet filter count{i}\n")

    print("Map")
    f.write("\nadd map inet filter testmap0 { type ipv4_addr : counter; flags interval;}\n\n")
    f.write("\nadd map inet filter testmap1 { type ipv4_addr : counter; flags interval;}\n\n")
    f.write("\nadd map inet filter testmap2 { type ipv4_addr : counter; flags interval;}\n\n")
    f.write("\nadd map inet filter testmap3 { type ipv4_addr : counter; flags interval;}\n\n")
    #f.write("\nadd map inet filter testmap0 { type ipv4_addr : counter;}\n\n")
    #f.write("\nadd map inet filter testmap1 { type ipv4_addr : counter;}\n\n")
    #f.write("\nadd map inet filter testmap2 { type ipv4_addr : counter;}\n\n")
    #f.write("\nadd map inet filter testmap3 { type ipv4_addr : counter;}\n\n")

    for i in range(256):
        for j in range(256):
            count = (i*256+j) % (32*256)
            map_number = count % 4
            f.write(f"add element inet filter testmap{map_number} {{ 10.0.{i}.{j} : \"count{count}\" }}\n")

Re: Issue with counter and interval map

[Pablo Neira Ayuso]

On Thu, Sep 14, 2023 at 12:52:24AM +0200, Pablo Neira Ayuso wrote:
> Hi Jann,
> 
> On Wed, Sep 13, 2023 at 07:42:01PM +0200, Jann Haber wrote:
> > Dear nftables-developers,
> > 
> > at Selfnet, we have been operating our CGN based on nftables for
> > roughly 4 years now (at that time we switched from iptables).
> > Recently, we have upgraded our first server from Debian bullseye
> > (Kernel 5.10, nftables 0.9.8) to bookworm (Kernel 6.1, nftables
> > 1.0.6). On bookworm, our ruleset that works well on bullseye fails
> > to load.
> > 
> > We have boiled it down to the minimal example attached, which fails
> > to load correctly on bookworm and also on a current Arch-Linux.
> > 
> > xxxxx@xxxxx:~$ sudo nft -f example.conf
> > example.conf:5:35-48: Error: Could not process rule: No such file or directory
> > add element inet filter testmap { 192.168.0.0/24 : "TEST" }
> >                                   ^^^^^^^^^^^^^^
> > What we have tested:
> > - Removing the last line from the file and running it later manually
> >   via the command line, there is no error
> > - Splitting the file in two (having the final line in a separate
> >   file), the two files can be applied with two nft -f calls with no
> >   error
> > - When swapping the lines 3 and 4 (i.e. first add counter, then add
> >   map), there is no error applying the file
> > - Removing "flags: interval" from the map and testing with a single
> >   IP, there is no error applying the file
> > 
> > In summary, I believe our rule syntax is ok - but something is going
> > wrong when the rules are applied in the given order atomically with
> > "nft -f". We appreciate any insight, please also let us know if we
> > did something wrong or if we can assist with debugging further.
> 
> I can reproduce it, this is a userspace bug which happens with
> interval sets in nft_cmd_post_expand(), I will post a fix asap.

Proposed patch to address this issue:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/20230916143549.57646-1-pablo@netfilter.org/

I am taking a look at the second issue you are reporting, I will
follow up.

Thanks.