Re: iptables-nft fails to restore huge rulesets

[Pablo Neira Ayuso <pablo@netfilter.org>]

On Thu, Apr 01, 2021 at 12:30:55PM +0200, Phil Sutter wrote:
> Hi,
> 
> On Wed, Mar 31, 2021 at 10:51:51PM +0200, Pablo Neira Ayuso wrote:
> > On Wed, Mar 31, 2021 at 04:41:40PM +0200, Phil Sutter wrote:
> > > On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote:
> > > > Phil Sutter <phil@nwl.cc> wrote:
> > > > > I'm currently trying to fix for an issue in Kubernetes realm[1]:
> > > > > Baseline is they are trying to restore a ruleset with ~700k lines and it
> > > > > fails. Needless to say, legacy iptables handles it just fine.
> > > > > 
> > > > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the
> > > > > batch to kernel, and this is what they're hitting.
> > > > > 
> > > > > I can work around that limit by increasing each iovec (via
> > > > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous:
> > > > > 
> > > > > With a scripted binary-search I checked the maximum working number of
> > > > > restore items of:
> > > > > 
> > > > > (1) User-defined chains
> > > > > (2) rules with merely comment match present
> > > > > (3) rules matching on saddr, daddr, iniface and outiface
> > > > > 
> > > > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE:
> > > > > 
> > > > > legacy		32 (stock)	  64		   128          256
> > > > > ----------------------------------------------------------------------
> > > > > 1'636'799	1'602'202	- NC -		  - NC -       - NC -
> > > > > 1'220'159	  302'079	604'160		1'208'320      - NC -
> > > > > 3'532'040	  242'688	485'376		  971'776    1'944'576
> > > > 
> > > > Can you explain that table? What does 1'636'799 mean? NC?
> > > 
> > > Ah, sorry: NC is "not care", I didn't consider those numbers relevant
> > > given that iptables-nft has caught up to legacy previously already.
> > > 
> > > 1'636'799 is the max number of user-defined chains I can successfully
> > > restore using iptables-legacy-restore. Looks like I dropped the rows'
> > > description while reformatting by accident: the first row of that table
> > > corresponds with test (1), second with test (2) and third with test (3).
> > > 
> > > So legacy may restore at once ~1.6M chains or ~1.2M comment rules or
> > > ~3.5M rules with {s,d}{addr,iface} matches.
> > > 
> > > The following columns are for iptables-nft with varying BATCH_PAGE_SIZE
> > > values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is
> > > 'N * getpagesize()' large.
> > 
> > Did you measure any slow down in the ruleset load time after selecting
> > a larger batch chunk size?
> 
> Restoring 100k rules shows no significant difference in between stock
> (32 * 8k) and 512 * 8k chunk sizes. So if you think it's acceptable to
> allocate 4MB of buffer at once, I'd just send a patch.

That's fine.

> Lifting that 1024 chunk count limit might be an alternative, but I guess
> that sits in kernel space?

That sits in the kernel, in the generic socket layer IIRC.

P.S: Would you mind to send a patch for nftables too to keep it in
sync? Thanks.