expose named set element count

expose named set element count

[Trevor Cordes]

Hi!  I have an application that can create very large named sets
(dynamically) of IPs.  For monitoring and testing purposes it is useful to
get a count of elements currently in a set.

Currently I an running "nft list table inet mytable |wc".  But once the ip
set grows to much over 100k IPs, that command can take around 90 minutes
on my system!  During that time the nft command takes 98% CPU.  That makes
monitoring in this manner nearly impossible.

I think the in-kernel code tracks the current number of elements.  Can we
expose this through the Netlink interface so users can get an instant
count of elements with virtually 0 CPU usage?  (Note, I'm not talking the
"size" data point, I'm talking the actual IPs currently in the set.)

Less importantly, I'm wondering what is making nft take so long to list
these huge sets.  Maybe the IP numeric sorting and formatting?  If it's
all the sorting, maybe an nft option to disable sorting would help?  Or an
option to disable formatting and have nft just dump the whole wad of IPs
on one line per set?

Thanks so much.  Please CC: my on replies because I'm not sure my
subscribe request is working or going anywhere.  The listserv may not like
my mail smarthost.

Re: expose named set element count

[Pablo Neira Ayuso]

On Tue, Oct 20, 2026 at 08:17:44AM -0500, Trevor Cordes wrote:
> Hi!  I have an application that can create very large named sets
> (dynamically) of IPs.  For monitoring and testing purposes it is useful to
> get a count of elements currently in a set.
> 
> Currently I an running "nft list table inet mytable |wc".  But once the ip
> set grows to much over 100k IPs, that command can take around 90 minutes
> on my system!  During that time the nft command takes 98% CPU.  That makes
> monitoring in this manner nearly impossible.
> 
> I think the in-kernel code tracks the current number of elements.  Can we
> expose this through the Netlink interface so users can get an instant
> count of elements with virtually 0 CPU usage?  (Note, I'm not talking the
> "size" data point, I'm talking the actual IPs currently in the set.)
> 
> Less importantly, I'm wondering what is making nft take so long to list
> these huge sets.  Maybe the IP numeric sorting and formatting?  If it's
> all the sorting, maybe an nft option to disable sorting would help?  Or an
> option to disable formatting and have nft just dump the whole wad of IPs
> on one line per set?
> 
> Thanks so much.  Please CC: my on replies because I'm not sure my
> subscribe request is working or going anywhere.  The listserv may not like
> my mail smarthost.

There is a count field in more recent userspace nft and kernels, what
nft and kernel versions are you running?

Re: expose named set element count

[Pablo Neira Ayuso]

On Tue, Oct 20, 2026 at 06:35:31PM +0200, Pablo Neira Ayuso wrote:
> On Tue, Oct 20, 2026 at 08:17:44AM -0500, Trevor Cordes wrote:
> > Hi!  I have an application that can create very large named sets
> > (dynamically) of IPs.  For monitoring and testing purposes it is useful to
> > get a count of elements currently in a set.
> > 
> > Currently I an running "nft list table inet mytable |wc".  But once the ip
> > set grows to much over 100k IPs, that command can take around 90 minutes
> > on my system!  During that time the nft command takes 98% CPU.  That makes
> > monitoring in this manner nearly impossible.
> > 
> > I think the in-kernel code tracks the current number of elements.  Can we
> > expose this through the Netlink interface so users can get an instant
> > count of elements with virtually 0 CPU usage?  (Note, I'm not talking the
> > "size" data point, I'm talking the actual IPs currently in the set.)
> > 
> > Less importantly, I'm wondering what is making nft take so long to list
> > these huge sets.  Maybe the IP numeric sorting and formatting?  If it's
> > all the sorting, maybe an nft option to disable sorting would help?  Or an
> > option to disable formatting and have nft just dump the whole wad of IPs
> > on one line per set?
> > 
> > Thanks so much.  Please CC: my on replies because I'm not sure my
> > subscribe request is working or going anywhere.  The listserv may not like
> > my mail smarthost.
> 
> There is a count field in more recent userspace nft and kernels, what
> nft and kernel versions are you running?

nft list ruleset
table ip t {
        set s {
                type ipv4_addr
                size 10 # count 1
                counter
                elements = { 1.1.1.1 counter packets 1 bytes 11 }
        }
}

Userspace supports is available since nftables 1.1.5:
$ git describe --contains 8ff87c8fd483eb3fdab5839cccaf100b593fe92e
v1.1.4~48

And Linux kernel needs to >= v6.17:
$ git describe --contains 0014af802193a
v6.16-rc1~132^2~190^2

Re: expose named set element count

[Trevor Cordes]

On 2026-10-20 Pablo Neira Ayuso wrote:
> On Tue, Oct 20, 2026 at 08:17:44AM -0500, Trevor Cordes wrote:
> > Hi!  I have an application that can create very large named sets
> > (dynamically) of IPs.  For monitoring and testing purposes it is
> > useful to get a count of elements currently in a set.
> > 
> > Currently I an running "nft list table inet mytable |wc".  But once
> > the ip set grows to much over 100k IPs, that command can take
> > around 90 minutes on my system!  During that time the nft command
> > takes 98% CPU.  That makes monitoring in this manner nearly
> > impossible.
>
> There is a count field in more recent userspace nft and kernels, what
> nft and kernel versions are you running?

Thanks for the quick response (and I got subscribed, yay).  That's
great news that the feature may already be there.

kernel 6.12.101 and nftables 1.1.3

Unfortunately we're stuck on slower-moving Debian 13 for this
production box, and from your version list it appears we may be too far
behind.

Unfortunately, I cannot easily
nft list ruleset
because that takes 90+ mins to run.  I could try overnight again.  I
don't recall seeing a "# count X" part to the output before, so I'm
guessing I'm too many versions behind?

The main thing though is that I can get that same size 10 # count 1
output WITHOUT running the full "ruleset", or even a subset -- it would
absolutely require a command I can run that *omits* the actual
elements.  Can the latest version do that?  If I run anything that
lists the 100's of k's of elements, the nft list command shows me
nothing for 90+ mins then dumps it all at once.

I am perfectly fine with waiting until Deb 14 or a newer 13 to get this
feature that might already be there.  My main concern is that I can
expect it sometime in the future, so I can turn my frown upside down.

Thanks!