Re: [PATCH nf,v2] netfilter: nft_set_rbtree: allocate same array size on updates

[Chris Arges <carges@cloudflare.com>]

On 2026-03-11 17:43:27, Pablo Neira Ayuso wrote:
> On Wed, Mar 11, 2026 at 11:29:59AM -0500, Chris Arges wrote:
> > On 2026-03-07 01:11:24, Pablo Neira Ayuso wrote:
> > > The array resize function increments the size of the array in
> > > NFT_ARRAY_EXTRA_SIZE slots for each update, this is unnecesarily
> > > increasing the array size.
> > > 
> > > To determine the number of array slots:
> > > 
> > > - Use NFT_ARRAY_EXTRA_SIZE for new sets.
> > > - Use the current maximum number of intervals in the live array.
> > > 
> > > Reported-by: Chris Arges <carges@cloudflare.com>
> > > Fixes: 7e43e0a1141d ("netfilter: nft_set_rbtree: translate rbtree to array for binary search")
> > > Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> > > ---
> > > v2: fix crash with new sets, reported by Florian.
> > > 
> > >  net/netfilter/nft_set_rbtree.c | 7 ++++++-
> > >  1 file changed, 6 insertions(+), 1 deletion(-)
> > > 
> > 
> > Pablo,
> 
> Chris,
> 
> > I was able to test with this patch applied on top of v6.18.16; however the
> > memory consumption remained high and similar to v6.18.16 without this patch.
> 
> Makes no sense to me.
> 

I did something like this:
```
git checkout v6.18.16
patch -p1 < this.patch
vng -b
```
Then ran my reproducer; unreclaimable slab memory went up past 4G showing
the high memory consumption.

> > My VM reproducer runs the services and checks for slab memory increases. In a
> > passing test case, the unreclaimable slab memory reaches about 1.4G and
> > stabilizes. In a failure test case unreclaimable slab memory goes beyond 4.4G
> > then the process gets OOM killed due to cgroup memory limits.
> 
> I have to review again what I posted. You mean, memory keeps for each
> dynamic update, increasing until it reaches OOM?
> 

Correct.

> > Also, using this reproducer I re-verified that this patch is what changes
> > slab memory.stat characteristics from within the cgroup:
> > - 7e43e0a1141d netfilter: nft_set_rbtree: translate rbtree to array for binary search
> > 
> > Finally, I reverted the following patches from v6.18.16 and re-tested using my
> > reproducer:
> > - 648946966a08 netfilter: nft_set_rbtree: validate open interval overlap
> > - 782f2688128e netfilter: nft_set_rbtree: validate element belonging to interval
> > - 35f83a75529a netfilter: nft_set_rbtree: don't gc elements on insert
> 
> Are you using timeout in your set?
>
No.
 
> > - 5599fa810b50 netfilter: nft_set_rbtree: remove seqcount_rwlock_t
> > - 2aa34191f06f netfilter: nft_set_rbtree: use binary search array in get command
> > - 7e43e0a1141d netfilter: nft_set_rbtree: translate rbtree to array for binary search
> > In this instance memory allocations were again around 1.4G.
> > 
> > Any suggestions for other tests, I can rebuild with memory debugging config as
> > well.
> > 
> > Also, could there be an option here to opt-out of this performance optimization
> > in favor of retaining existing memory characteristics?
> 
> This series is fixing a real bug, now you may experience possible
> false negatives in lookups with what you have reverted.
> 

Understood, we want to use these patches in the long run; once memory usage
issues are fixed.

> I am going to collect memory numbers and post them here, I will try to
> mimic a script from your description.

Sure, I can provide more details if needed, but should be similar as
explained in https://lore.kernel.org/all/aahw_h5DdmYZeeqw@20HS2G4/.

Thanks,
--chris