Re: Issue with counter and interval map

[Jann Haber <jannh@selfnet.de>]

[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]

Hi Pablo,

thank you first of all for your prompt response, very appreciated!

We have a second issue also related to interval maps and counters, however this seems to be more along the lines of a userspace performance problem. At Selfnet, we will soon hit 7000 members, with our traffic running across two redundant CGN machines powered by nftables. Naturally, our maps and sets can become quite large and may have some tens of thousands of entries.

I have attached a small python script that creates an "example.conf" file containing a simple set of very repetitive rules including counters and 4 interval maps each of the size ~16000 entries referencing these counters. On our bookworm and ArchLinux hosts, the resulting rules take very long to load with "nft -f" (at least multiple minutes). All other maps in our ruleset, some of which have similar size, are loaded instantly, so the size itself seems not to be an issue for nft in general.
If needed I can also provide the example.conf file itself, but due to the size of a few MB, I rather avoid to spam the mailing list.

Further info:
- With a regular map instead of an interval map (just remove the "flags interval" in the example), the rules are loaded in fractions of a second
- Using add map { [...] elements = [...] } instead of add map { [...] }; add element [...]; add element [...] and so on, the ruleset is also loaded quickly (this is our workaround that we use for now)
- We have had no issues with this kind of ruleset on Debian Bullseye (Kernel 5.10, nftables 0.9.8), it seems to have been introduced later

Thanks again and looking forward to your feedback!

Kind Regards,
Jann

[-- Attachment #2: nftables-slow.py --]
[-- Type: text/x-python, Size: 1199 bytes --]



with open("example.conf", "w") as f:
    print("Initial lines")
    f.write("flush ruleset\n")
    f.write("add table inet filter\n")

    print("Counters")
    for i in range(32*256):
        f.write(f"add counter inet filter count{i}\n")

    print("Map")
    f.write("\nadd map inet filter testmap0 { type ipv4_addr : counter; flags interval;}\n\n")
    f.write("\nadd map inet filter testmap1 { type ipv4_addr : counter; flags interval;}\n\n")
    f.write("\nadd map inet filter testmap2 { type ipv4_addr : counter; flags interval;}\n\n")
    f.write("\nadd map inet filter testmap3 { type ipv4_addr : counter; flags interval;}\n\n")
    #f.write("\nadd map inet filter testmap0 { type ipv4_addr : counter;}\n\n")
    #f.write("\nadd map inet filter testmap1 { type ipv4_addr : counter;}\n\n")
    #f.write("\nadd map inet filter testmap2 { type ipv4_addr : counter;}\n\n")
    #f.write("\nadd map inet filter testmap3 { type ipv4_addr : counter;}\n\n")

    for i in range(256):
        for j in range(256):
            count = (i*256+j) % (32*256)
            map_number = count % 4
            f.write(f"add element inet filter testmap{map_number} {{ 10.0.{i}.{j} : \"count{count}\" }}\n")