Re: [PATCH nf,v2] netfilter: nft_set_rbtree: allocate same array size on updates

[Pablo Neira Ayuso <pablo@netfilter.org>]

On Wed, Mar 11, 2026 at 11:29:59AM -0500, Chris Arges wrote:
> On 2026-03-07 01:11:24, Pablo Neira Ayuso wrote:
> > The array resize function increments the size of the array in
> > NFT_ARRAY_EXTRA_SIZE slots for each update, this is unnecesarily
> > increasing the array size.
> > 
> > To determine the number of array slots:
> > 
> > - Use NFT_ARRAY_EXTRA_SIZE for new sets.
> > - Use the current maximum number of intervals in the live array.
> > 
> > Reported-by: Chris Arges <carges@cloudflare.com>
> > Fixes: 7e43e0a1141d ("netfilter: nft_set_rbtree: translate rbtree to array for binary search")
> > Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> > ---
> > v2: fix crash with new sets, reported by Florian.
> > 
> >  net/netfilter/nft_set_rbtree.c | 7 ++++++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> > 
> 
> Pablo,

Chris,

> I was able to test with this patch applied on top of v6.18.16; however the
> memory consumption remained high and similar to v6.18.16 without this patch.

Makes no sense to me.

> My VM reproducer runs the services and checks for slab memory increases. In a
> passing test case, the unreclaimable slab memory reaches about 1.4G and
> stabilizes. In a failure test case unreclaimable slab memory goes beyond 4.4G
> then the process gets OOM killed due to cgroup memory limits.

I have to review again what I posted. You mean, memory keeps for each
dynamic update, increasing until it reaches OOM?

> Also, using this reproducer I re-verified that this patch is what changes
> slab memory.stat characteristics from within the cgroup:
> - 7e43e0a1141d netfilter: nft_set_rbtree: translate rbtree to array for binary search
> 
> Finally, I reverted the following patches from v6.18.16 and re-tested using my
> reproducer:
> - 648946966a08 netfilter: nft_set_rbtree: validate open interval overlap
> - 782f2688128e netfilter: nft_set_rbtree: validate element belonging to interval
> - 35f83a75529a netfilter: nft_set_rbtree: don't gc elements on insert

Are you using timeout in your set?

> - 5599fa810b50 netfilter: nft_set_rbtree: remove seqcount_rwlock_t
> - 2aa34191f06f netfilter: nft_set_rbtree: use binary search array in get command
> - 7e43e0a1141d netfilter: nft_set_rbtree: translate rbtree to array for binary search
> In this instance memory allocations were again around 1.4G.
> 
> Any suggestions for other tests, I can rebuild with memory debugging config as
> well.
> 
> Also, could there be an option here to opt-out of this performance optimization
> in favor of retaining existing memory characteristics?

This series is fixing a real bug, now you may experience possible
false negatives in lookups with what you have reverted.

I am going to collect memory numbers and post them here, I will try to
mimic a script from your description.