* iptables-nft fails to restore huge rulesets
@ 2021-03-31 9:13 Phil Sutter
2021-03-31 13:35 ` Florian Westphal
0 siblings, 1 reply; 6+ messages in thread
From: Phil Sutter @ 2021-03-31 9:13 UTC (permalink / raw)
To: Pablo Neira Ayuso, Florian Westphal; +Cc: netfilter-devel
Hi,
I'm currently trying to fix for an issue in Kubernetes realm[1]:
Baseline is they are trying to restore a ruleset with ~700k lines and it
fails. Needless to say, legacy iptables handles it just fine.
Meanwhile I found out there's a limit of 1024 iovecs when submitting the
batch to kernel, and this is what they're hitting.
I can work around that limit by increasing each iovec (via
BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous:
With a scripted binary-search I checked the maximum working number of
restore items of:
(1) User-defined chains
(2) rules with merely comment match present
(3) rules matching on saddr, daddr, iniface and outiface
Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE:
legacy 32 (stock) 64 128 256
----------------------------------------------------------------------
1'636'799 1'602'202 - NC - - NC - - NC -
1'220'159 302'079 604'160 1'208'320 - NC -
3'532'040 242'688 485'376 971'776 1'944'576
At this point I stopped as the VM's 20GB RAM became the limit
(iptables-nft-restore being OOM-killed instead of just failing).
What would you suggest? Should I just change BATCH_PAGE_SIZE to make it
"large enough" or is there a better approach?
Cheers, Phil
[1] https://github.com/kubernetes/kubernetes/issues/96018
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets
2021-03-31 9:13 iptables-nft fails to restore huge rulesets Phil Sutter
@ 2021-03-31 13:35 ` Florian Westphal
2021-03-31 14:41 ` Phil Sutter
0 siblings, 1 reply; 6+ messages in thread
From: Florian Westphal @ 2021-03-31 13:35 UTC (permalink / raw)
To: Phil Sutter, Pablo Neira Ayuso, Florian Westphal, netfilter-devel
Phil Sutter <phil@nwl.cc> wrote:
> Hi,
>
> I'm currently trying to fix for an issue in Kubernetes realm[1]:
> Baseline is they are trying to restore a ruleset with ~700k lines and it
> fails. Needless to say, legacy iptables handles it just fine.
>
> Meanwhile I found out there's a limit of 1024 iovecs when submitting the
> batch to kernel, and this is what they're hitting.
>
> I can work around that limit by increasing each iovec (via
> BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous:
>
> With a scripted binary-search I checked the maximum working number of
> restore items of:
>
> (1) User-defined chains
> (2) rules with merely comment match present
> (3) rules matching on saddr, daddr, iniface and outiface
>
> Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE:
>
> legacy 32 (stock) 64 128 256
> ----------------------------------------------------------------------
> 1'636'799 1'602'202 - NC - - NC - - NC -
> 1'220'159 302'079 604'160 1'208'320 - NC -
> 3'532'040 242'688 485'376 971'776 1'944'576
Can you explain that table? What does 1'636'799 mean? NC?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets
2021-03-31 13:35 ` Florian Westphal
@ 2021-03-31 14:41 ` Phil Sutter
2021-03-31 20:51 ` Pablo Neira Ayuso
0 siblings, 1 reply; 6+ messages in thread
From: Phil Sutter @ 2021-03-31 14:41 UTC (permalink / raw)
To: Florian Westphal; +Cc: Pablo Neira Ayuso, netfilter-devel
On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote:
> Phil Sutter <phil@nwl.cc> wrote:
> > I'm currently trying to fix for an issue in Kubernetes realm[1]:
> > Baseline is they are trying to restore a ruleset with ~700k lines and it
> > fails. Needless to say, legacy iptables handles it just fine.
> >
> > Meanwhile I found out there's a limit of 1024 iovecs when submitting the
> > batch to kernel, and this is what they're hitting.
> >
> > I can work around that limit by increasing each iovec (via
> > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous:
> >
> > With a scripted binary-search I checked the maximum working number of
> > restore items of:
> >
> > (1) User-defined chains
> > (2) rules with merely comment match present
> > (3) rules matching on saddr, daddr, iniface and outiface
> >
> > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE:
> >
> > legacy 32 (stock) 64 128 256
> > ----------------------------------------------------------------------
> > 1'636'799 1'602'202 - NC - - NC - - NC -
> > 1'220'159 302'079 604'160 1'208'320 - NC -
> > 3'532'040 242'688 485'376 971'776 1'944'576
>
> Can you explain that table? What does 1'636'799 mean? NC?
Ah, sorry: NC is "not care", I didn't consider those numbers relevant
given that iptables-nft has caught up to legacy previously already.
1'636'799 is the max number of user-defined chains I can successfully
restore using iptables-legacy-restore. Looks like I dropped the rows'
description while reformatting by accident: the first row of that table
corresponds with test (1), second with test (2) and third with test (3).
So legacy may restore at once ~1.6M chains or ~1.2M comment rules or
~3.5M rules with {s,d}{addr,iface} matches.
The following columns are for iptables-nft with varying BATCH_PAGE_SIZE
values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is
'N * getpagesize()' large.
Cheers, Phil
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets
2021-03-31 14:41 ` Phil Sutter
@ 2021-03-31 20:51 ` Pablo Neira Ayuso
2021-04-01 10:30 ` Phil Sutter
0 siblings, 1 reply; 6+ messages in thread
From: Pablo Neira Ayuso @ 2021-03-31 20:51 UTC (permalink / raw)
To: Phil Sutter, Florian Westphal, netfilter-devel
On Wed, Mar 31, 2021 at 04:41:40PM +0200, Phil Sutter wrote:
> On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote:
> > Phil Sutter <phil@nwl.cc> wrote:
> > > I'm currently trying to fix for an issue in Kubernetes realm[1]:
> > > Baseline is they are trying to restore a ruleset with ~700k lines and it
> > > fails. Needless to say, legacy iptables handles it just fine.
> > >
> > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the
> > > batch to kernel, and this is what they're hitting.
> > >
> > > I can work around that limit by increasing each iovec (via
> > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous:
> > >
> > > With a scripted binary-search I checked the maximum working number of
> > > restore items of:
> > >
> > > (1) User-defined chains
> > > (2) rules with merely comment match present
> > > (3) rules matching on saddr, daddr, iniface and outiface
> > >
> > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE:
> > >
> > > legacy 32 (stock) 64 128 256
> > > ----------------------------------------------------------------------
> > > 1'636'799 1'602'202 - NC - - NC - - NC -
> > > 1'220'159 302'079 604'160 1'208'320 - NC -
> > > 3'532'040 242'688 485'376 971'776 1'944'576
> >
> > Can you explain that table? What does 1'636'799 mean? NC?
>
> Ah, sorry: NC is "not care", I didn't consider those numbers relevant
> given that iptables-nft has caught up to legacy previously already.
>
> 1'636'799 is the max number of user-defined chains I can successfully
> restore using iptables-legacy-restore. Looks like I dropped the rows'
> description while reformatting by accident: the first row of that table
> corresponds with test (1), second with test (2) and third with test (3).
>
> So legacy may restore at once ~1.6M chains or ~1.2M comment rules or
> ~3.5M rules with {s,d}{addr,iface} matches.
>
> The following columns are for iptables-nft with varying BATCH_PAGE_SIZE
> values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is
> 'N * getpagesize()' large.
Did you measure any slow down in the ruleset load time after selecting
a larger batch chunk size?
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets
2021-03-31 20:51 ` Pablo Neira Ayuso
@ 2021-04-01 10:30 ` Phil Sutter
2021-04-01 11:44 ` Pablo Neira Ayuso
0 siblings, 1 reply; 6+ messages in thread
From: Phil Sutter @ 2021-04-01 10:30 UTC (permalink / raw)
To: Pablo Neira Ayuso; +Cc: Florian Westphal, netfilter-devel
Hi,
On Wed, Mar 31, 2021 at 10:51:51PM +0200, Pablo Neira Ayuso wrote:
> On Wed, Mar 31, 2021 at 04:41:40PM +0200, Phil Sutter wrote:
> > On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote:
> > > Phil Sutter <phil@nwl.cc> wrote:
> > > > I'm currently trying to fix for an issue in Kubernetes realm[1]:
> > > > Baseline is they are trying to restore a ruleset with ~700k lines and it
> > > > fails. Needless to say, legacy iptables handles it just fine.
> > > >
> > > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the
> > > > batch to kernel, and this is what they're hitting.
> > > >
> > > > I can work around that limit by increasing each iovec (via
> > > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous:
> > > >
> > > > With a scripted binary-search I checked the maximum working number of
> > > > restore items of:
> > > >
> > > > (1) User-defined chains
> > > > (2) rules with merely comment match present
> > > > (3) rules matching on saddr, daddr, iniface and outiface
> > > >
> > > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE:
> > > >
> > > > legacy 32 (stock) 64 128 256
> > > > ----------------------------------------------------------------------
> > > > 1'636'799 1'602'202 - NC - - NC - - NC -
> > > > 1'220'159 302'079 604'160 1'208'320 - NC -
> > > > 3'532'040 242'688 485'376 971'776 1'944'576
> > >
> > > Can you explain that table? What does 1'636'799 mean? NC?
> >
> > Ah, sorry: NC is "not care", I didn't consider those numbers relevant
> > given that iptables-nft has caught up to legacy previously already.
> >
> > 1'636'799 is the max number of user-defined chains I can successfully
> > restore using iptables-legacy-restore. Looks like I dropped the rows'
> > description while reformatting by accident: the first row of that table
> > corresponds with test (1), second with test (2) and third with test (3).
> >
> > So legacy may restore at once ~1.6M chains or ~1.2M comment rules or
> > ~3.5M rules with {s,d}{addr,iface} matches.
> >
> > The following columns are for iptables-nft with varying BATCH_PAGE_SIZE
> > values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is
> > 'N * getpagesize()' large.
>
> Did you measure any slow down in the ruleset load time after selecting
> a larger batch chunk size?
Restoring 100k rules shows no significant difference in between stock
(32 * 8k) and 512 * 8k chunk sizes. So if you think it's acceptable to
allocate 4MB of buffer at once, I'd just send a patch.
Lifting that 1024 chunk count limit might be an alternative, but I guess
that sits in kernel space?
Cheers, Phil
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets
2021-04-01 10:30 ` Phil Sutter
@ 2021-04-01 11:44 ` Pablo Neira Ayuso
0 siblings, 0 replies; 6+ messages in thread
From: Pablo Neira Ayuso @ 2021-04-01 11:44 UTC (permalink / raw)
To: Phil Sutter, Florian Westphal, netfilter-devel
On Thu, Apr 01, 2021 at 12:30:55PM +0200, Phil Sutter wrote:
> Hi,
>
> On Wed, Mar 31, 2021 at 10:51:51PM +0200, Pablo Neira Ayuso wrote:
> > On Wed, Mar 31, 2021 at 04:41:40PM +0200, Phil Sutter wrote:
> > > On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote:
> > > > Phil Sutter <phil@nwl.cc> wrote:
> > > > > I'm currently trying to fix for an issue in Kubernetes realm[1]:
> > > > > Baseline is they are trying to restore a ruleset with ~700k lines and it
> > > > > fails. Needless to say, legacy iptables handles it just fine.
> > > > >
> > > > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the
> > > > > batch to kernel, and this is what they're hitting.
> > > > >
> > > > > I can work around that limit by increasing each iovec (via
> > > > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous:
> > > > >
> > > > > With a scripted binary-search I checked the maximum working number of
> > > > > restore items of:
> > > > >
> > > > > (1) User-defined chains
> > > > > (2) rules with merely comment match present
> > > > > (3) rules matching on saddr, daddr, iniface and outiface
> > > > >
> > > > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE:
> > > > >
> > > > > legacy 32 (stock) 64 128 256
> > > > > ----------------------------------------------------------------------
> > > > > 1'636'799 1'602'202 - NC - - NC - - NC -
> > > > > 1'220'159 302'079 604'160 1'208'320 - NC -
> > > > > 3'532'040 242'688 485'376 971'776 1'944'576
> > > >
> > > > Can you explain that table? What does 1'636'799 mean? NC?
> > >
> > > Ah, sorry: NC is "not care", I didn't consider those numbers relevant
> > > given that iptables-nft has caught up to legacy previously already.
> > >
> > > 1'636'799 is the max number of user-defined chains I can successfully
> > > restore using iptables-legacy-restore. Looks like I dropped the rows'
> > > description while reformatting by accident: the first row of that table
> > > corresponds with test (1), second with test (2) and third with test (3).
> > >
> > > So legacy may restore at once ~1.6M chains or ~1.2M comment rules or
> > > ~3.5M rules with {s,d}{addr,iface} matches.
> > >
> > > The following columns are for iptables-nft with varying BATCH_PAGE_SIZE
> > > values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is
> > > 'N * getpagesize()' large.
> >
> > Did you measure any slow down in the ruleset load time after selecting
> > a larger batch chunk size?
>
> Restoring 100k rules shows no significant difference in between stock
> (32 * 8k) and 512 * 8k chunk sizes. So if you think it's acceptable to
> allocate 4MB of buffer at once, I'd just send a patch.
That's fine.
> Lifting that 1024 chunk count limit might be an alternative, but I guess
> that sits in kernel space?
That sits in the kernel, in the generic socket layer IIRC.
P.S: Would you mind to send a patch for nftables too to keep it in
sync? Thanks.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-04-01 17:58 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-03-31 9:13 iptables-nft fails to restore huge rulesets Phil Sutter
2021-03-31 13:35 ` Florian Westphal
2021-03-31 14:41 ` Phil Sutter
2021-03-31 20:51 ` Pablo Neira Ayuso
2021-04-01 10:30 ` Phil Sutter
2021-04-01 11:44 ` Pablo Neira Ayuso
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).