Re: [PATCH nf,v4] netfilter: nft_set_rbtree: revisit array resize logic

[Pablo Neira Ayuso <pablo@netfilter.org>]

On Wed, Mar 25, 2026 at 07:44:36PM -0500, Chris Arges wrote:
> On 2026-03-25 19:42:54, Chris Arges wrote:
> > On 2026-03-23 12:29:30, Chris Arges wrote:
> > > On 2026-03-18 17:57:54, Pablo Neira Ayuso wrote:
> > > > On Wed, Mar 18, 2026 at 10:46:56AM -0500, Chris Arges wrote:
> > > > > On 2026-03-17 18:07:21, Pablo Neira Ayuso wrote:
> > > > > > Chris Arges reports high memory consumption with thousands of
> > > > > > containers, this patch revisits the array allocation logic.
> > > > > > 
> > > > > > For anonymous sets, start by 16 slots (which takes 256 bytes on x86_64).
> > > > > > Expand it by x2 until threshold of 512 slots is reached, over that
> > > > > > threshold, expand it by x1.5.
> > > > > > 
> > > > > > For non-anonymous set, start by 1024 slots in the array (which takes 16
> > > > > > Kbytes initially on x86_64). Expand it by x1.5.
> > > > > > 
> > > > > > Use set->ndeact to subtract deactivated elements when calculating the
> > > > > > number of the slots in the array, otherwise the array size array gets
> > > > > > increased artifically. Add special case shrink logic to deal with flush
> > > > > > set too.
> > > > > > 
> > > > > > The shrink logic is skipped by anonymous sets.
> > > > > > 
> > > > > > Use check_add_overflow() to calculate the new array size.
> > > > > > 
> > > > > > Add a WARN_ON_ONCE check to make sure elements fit into the new array
> > > > > > size.
> > > > > > 
> > > > > > Reported-by: Chris Arges <carges@cloudflare.com>
> > > > > > Fixes: 7e43e0a1141d ("netfilter: nft_set_rbtree: translate rbtree to array for binary search")
> > > > > > Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
> > > > > > ---
> > > > > > v4: use maybe_grow: goto tag instead of grow:
> > > > > >     Add note in commit description: "The shrink logic is skipped by anonymous sets."
> > > > > > 
> > > > > 
> > > > > I will be able to testing this more in depth early next week. Just to confirm,
> > > > > this patch requires this to be applied first?
> > > > > https://lore.kernel.org/netfilter-devel/20260307001124.2897063-1-pablo@netfilter.org/
> > > > 
> > > > Yes, it requires that fix to be applied first.
> > > 
> > > Thanks, these two patches applied on 6.18 stable show slab unreclaimable memory
> > > leveling out at 1.5GB for my local reproducer. I'll be deploying this to a
> > > wider set of machines for more real-world testing this week.
> > > 
> > > So far seems good.
> > > --chris
> > 
> > We have deployed this successfully and memory usage looks good in production.
> > 
> > Tested-by: Chris Arges <carges@cloudflare.com>
> > 
> > --chris
> 
> Oh I see it's queued up here:
> https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260325222615.637793-8-pablo@netfilter.org/
> Thanks for fixing this!

Yes, I did not get in time to add your Tested-by: tag.

Thanks for testing in any case!