Re: Sets update

["Kerin Millar" <kfm@plushkava.net>]

On Sun, 21 Jul 2024, at 8:44 AM, Slavko wrote:
> Hi all,
>
> i recently start to port my ipsets to nftables set. I was using these
> ipsets (beside other) in manner, where they was some timeout set
> and its content was regulary updated (from various sources, online
> and local). If some IP(v6) was removed, i didn't bother to remove it
> from ipset, as it was removed by timeout...
>
> Now i fight with the same approach in nftables sets (kernel 5.15, nft
> 1.0.8). I learned, that to update element's timeout i need to remove
> element and then (re)add it again. As here is not simple way to do
> it from shell (simple shell command) i play with python script, which
> consumes IP list on input and produces appropriate nft commands,
> eg.:
>
>    fetch someiplist | myscript.py | nft -f -
>
> Now i am not sure, how to produce that output. Have i do it per IP?
> Eg.:
>
>     add element ... {IP1}
>     delete element ... {IP1}
>     add element ... {IP1 ...}
>     add element ... {IP2}
>     delete element ... {IP2}
>     add element ... {IP2 ...}
>     etc

Firstly, you should not use "delete element". It will never be reliable because a given element may have expired by the time the command is processed, in which case nft will raise an error and abort. Use "destroy element" instead.

Secondly, though you could handle each element one by one, I would not recommend it.

>
> Or have i produce output for all IPs at once? Eg:
>
>     add element ... {IP1, IP2, ...}
>     delete element ... {IP1, IP2, ...}
>     add element ... {IP1 ..., IP2 ..., ...}

Yes, grouping elements in this way is sensible. Note that there is no need to worry about the ARG_MAX constraint because nft is being made to read from the standard input rather than use the -c option. However, I would suggest generating one element per line because the way in which nft reports errors is atrocious in the case of long lines.

{
   echo "add element ... {"
   sed 's/$/,/'
   echo "}"
} | nft -f -

>
> Please, is here technical difference or something other to consider?
> The IP lists ranges from some hundreds to some thounsand of IPs,
> thus nothing really big, but not small.
>
> Another question/problem is, that this approach (delete/add) does't
> preserve counters. Please, how to preserve counters? Is only way
> to fetch and parse counters before i delete element and then add
> them into final add? IMO, this isn't very memory friendly (in script...).

As far as I am aware, it is impossible to add elements in a way that merely reset their expiry times in the case that they exist, which is a ridiculous state of affairs. Though there exists the "update" set statement, only rules can take advantage of it. Neither of the following commands achieve the intended outcome, despite being processed without error.

add element ... { <existing-ip> timeout <new-timeout> }
add element ... { <existing-ip> expires <new-timeout> }

>
> While these counters are not crucial for me, i use them in some
> ipsets for statistics, eg. i fill items from various sources into one
> ipset and group/count them by comment then. In ipset it is really
> simple task for awk and it was working even on small OpenWrt
> devices. But nftables sets doesn't produce as straighforward output.
> Please, how i can/have to parse element' counters? I am even not
> sure, if i will able to parse it by python (but i didn't try it yet), is here
> some tool for that?

I doubt it. Perhaps we need a new "update element" command to act similarly to the set statement bearing the same name.

Mind you, it gets worse. While the nft utility supports a JSON output mode to ease parsing, its output is less accurate than the conventional output mode in so far as all "timeout" and "expires" values are truncated to integer seconds and conveyed as JSON integers.

-- 
Kerin Millar