* iptables-nft fails to restore huge rulesets @ 2021-03-31 9:13 Phil Sutter 2021-03-31 13:35 ` Florian Westphal 0 siblings, 1 reply; 6+ messages in thread From: Phil Sutter @ 2021-03-31 9:13 UTC (permalink / raw) To: Pablo Neira Ayuso, Florian Westphal; +Cc: netfilter-devel Hi, I'm currently trying to fix for an issue in Kubernetes realm[1]: Baseline is they are trying to restore a ruleset with ~700k lines and it fails. Needless to say, legacy iptables handles it just fine. Meanwhile I found out there's a limit of 1024 iovecs when submitting the batch to kernel, and this is what they're hitting. I can work around that limit by increasing each iovec (via BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous: With a scripted binary-search I checked the maximum working number of restore items of: (1) User-defined chains (2) rules with merely comment match present (3) rules matching on saddr, daddr, iniface and outiface Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE: legacy 32 (stock) 64 128 256 ---------------------------------------------------------------------- 1'636'799 1'602'202 - NC - - NC - - NC - 1'220'159 302'079 604'160 1'208'320 - NC - 3'532'040 242'688 485'376 971'776 1'944'576 At this point I stopped as the VM's 20GB RAM became the limit (iptables-nft-restore being OOM-killed instead of just failing). What would you suggest? Should I just change BATCH_PAGE_SIZE to make it "large enough" or is there a better approach? Cheers, Phil [1] https://github.com/kubernetes/kubernetes/issues/96018 ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets 2021-03-31 9:13 iptables-nft fails to restore huge rulesets Phil Sutter @ 2021-03-31 13:35 ` Florian Westphal 2021-03-31 14:41 ` Phil Sutter 0 siblings, 1 reply; 6+ messages in thread From: Florian Westphal @ 2021-03-31 13:35 UTC (permalink / raw) To: Phil Sutter, Pablo Neira Ayuso, Florian Westphal, netfilter-devel Phil Sutter <phil@nwl.cc> wrote: > Hi, > > I'm currently trying to fix for an issue in Kubernetes realm[1]: > Baseline is they are trying to restore a ruleset with ~700k lines and it > fails. Needless to say, legacy iptables handles it just fine. > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the > batch to kernel, and this is what they're hitting. > > I can work around that limit by increasing each iovec (via > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous: > > With a scripted binary-search I checked the maximum working number of > restore items of: > > (1) User-defined chains > (2) rules with merely comment match present > (3) rules matching on saddr, daddr, iniface and outiface > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE: > > legacy 32 (stock) 64 128 256 > ---------------------------------------------------------------------- > 1'636'799 1'602'202 - NC - - NC - - NC - > 1'220'159 302'079 604'160 1'208'320 - NC - > 3'532'040 242'688 485'376 971'776 1'944'576 Can you explain that table? What does 1'636'799 mean? NC? ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets 2021-03-31 13:35 ` Florian Westphal @ 2021-03-31 14:41 ` Phil Sutter 2021-03-31 20:51 ` Pablo Neira Ayuso 0 siblings, 1 reply; 6+ messages in thread From: Phil Sutter @ 2021-03-31 14:41 UTC (permalink / raw) To: Florian Westphal; +Cc: Pablo Neira Ayuso, netfilter-devel On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote: > Phil Sutter <phil@nwl.cc> wrote: > > I'm currently trying to fix for an issue in Kubernetes realm[1]: > > Baseline is they are trying to restore a ruleset with ~700k lines and it > > fails. Needless to say, legacy iptables handles it just fine. > > > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the > > batch to kernel, and this is what they're hitting. > > > > I can work around that limit by increasing each iovec (via > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous: > > > > With a scripted binary-search I checked the maximum working number of > > restore items of: > > > > (1) User-defined chains > > (2) rules with merely comment match present > > (3) rules matching on saddr, daddr, iniface and outiface > > > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE: > > > > legacy 32 (stock) 64 128 256 > > ---------------------------------------------------------------------- > > 1'636'799 1'602'202 - NC - - NC - - NC - > > 1'220'159 302'079 604'160 1'208'320 - NC - > > 3'532'040 242'688 485'376 971'776 1'944'576 > > Can you explain that table? What does 1'636'799 mean? NC? Ah, sorry: NC is "not care", I didn't consider those numbers relevant given that iptables-nft has caught up to legacy previously already. 1'636'799 is the max number of user-defined chains I can successfully restore using iptables-legacy-restore. Looks like I dropped the rows' description while reformatting by accident: the first row of that table corresponds with test (1), second with test (2) and third with test (3). So legacy may restore at once ~1.6M chains or ~1.2M comment rules or ~3.5M rules with {s,d}{addr,iface} matches. The following columns are for iptables-nft with varying BATCH_PAGE_SIZE values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is 'N * getpagesize()' large. Cheers, Phil ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets 2021-03-31 14:41 ` Phil Sutter @ 2021-03-31 20:51 ` Pablo Neira Ayuso 2021-04-01 10:30 ` Phil Sutter 0 siblings, 1 reply; 6+ messages in thread From: Pablo Neira Ayuso @ 2021-03-31 20:51 UTC (permalink / raw) To: Phil Sutter, Florian Westphal, netfilter-devel On Wed, Mar 31, 2021 at 04:41:40PM +0200, Phil Sutter wrote: > On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote: > > Phil Sutter <phil@nwl.cc> wrote: > > > I'm currently trying to fix for an issue in Kubernetes realm[1]: > > > Baseline is they are trying to restore a ruleset with ~700k lines and it > > > fails. Needless to say, legacy iptables handles it just fine. > > > > > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the > > > batch to kernel, and this is what they're hitting. > > > > > > I can work around that limit by increasing each iovec (via > > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous: > > > > > > With a scripted binary-search I checked the maximum working number of > > > restore items of: > > > > > > (1) User-defined chains > > > (2) rules with merely comment match present > > > (3) rules matching on saddr, daddr, iniface and outiface > > > > > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE: > > > > > > legacy 32 (stock) 64 128 256 > > > ---------------------------------------------------------------------- > > > 1'636'799 1'602'202 - NC - - NC - - NC - > > > 1'220'159 302'079 604'160 1'208'320 - NC - > > > 3'532'040 242'688 485'376 971'776 1'944'576 > > > > Can you explain that table? What does 1'636'799 mean? NC? > > Ah, sorry: NC is "not care", I didn't consider those numbers relevant > given that iptables-nft has caught up to legacy previously already. > > 1'636'799 is the max number of user-defined chains I can successfully > restore using iptables-legacy-restore. Looks like I dropped the rows' > description while reformatting by accident: the first row of that table > corresponds with test (1), second with test (2) and third with test (3). > > So legacy may restore at once ~1.6M chains or ~1.2M comment rules or > ~3.5M rules with {s,d}{addr,iface} matches. > > The following columns are for iptables-nft with varying BATCH_PAGE_SIZE > values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is > 'N * getpagesize()' large. Did you measure any slow down in the ruleset load time after selecting a larger batch chunk size? ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets 2021-03-31 20:51 ` Pablo Neira Ayuso @ 2021-04-01 10:30 ` Phil Sutter 2021-04-01 11:44 ` Pablo Neira Ayuso 0 siblings, 1 reply; 6+ messages in thread From: Phil Sutter @ 2021-04-01 10:30 UTC (permalink / raw) To: Pablo Neira Ayuso; +Cc: Florian Westphal, netfilter-devel Hi, On Wed, Mar 31, 2021 at 10:51:51PM +0200, Pablo Neira Ayuso wrote: > On Wed, Mar 31, 2021 at 04:41:40PM +0200, Phil Sutter wrote: > > On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote: > > > Phil Sutter <phil@nwl.cc> wrote: > > > > I'm currently trying to fix for an issue in Kubernetes realm[1]: > > > > Baseline is they are trying to restore a ruleset with ~700k lines and it > > > > fails. Needless to say, legacy iptables handles it just fine. > > > > > > > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the > > > > batch to kernel, and this is what they're hitting. > > > > > > > > I can work around that limit by increasing each iovec (via > > > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous: > > > > > > > > With a scripted binary-search I checked the maximum working number of > > > > restore items of: > > > > > > > > (1) User-defined chains > > > > (2) rules with merely comment match present > > > > (3) rules matching on saddr, daddr, iniface and outiface > > > > > > > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE: > > > > > > > > legacy 32 (stock) 64 128 256 > > > > ---------------------------------------------------------------------- > > > > 1'636'799 1'602'202 - NC - - NC - - NC - > > > > 1'220'159 302'079 604'160 1'208'320 - NC - > > > > 3'532'040 242'688 485'376 971'776 1'944'576 > > > > > > Can you explain that table? What does 1'636'799 mean? NC? > > > > Ah, sorry: NC is "not care", I didn't consider those numbers relevant > > given that iptables-nft has caught up to legacy previously already. > > > > 1'636'799 is the max number of user-defined chains I can successfully > > restore using iptables-legacy-restore. Looks like I dropped the rows' > > description while reformatting by accident: the first row of that table > > corresponds with test (1), second with test (2) and third with test (3). > > > > So legacy may restore at once ~1.6M chains or ~1.2M comment rules or > > ~3.5M rules with {s,d}{addr,iface} matches. > > > > The following columns are for iptables-nft with varying BATCH_PAGE_SIZE > > values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is > > 'N * getpagesize()' large. > > Did you measure any slow down in the ruleset load time after selecting > a larger batch chunk size? Restoring 100k rules shows no significant difference in between stock (32 * 8k) and 512 * 8k chunk sizes. So if you think it's acceptable to allocate 4MB of buffer at once, I'd just send a patch. Lifting that 1024 chunk count limit might be an alternative, but I guess that sits in kernel space? Cheers, Phil ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: iptables-nft fails to restore huge rulesets 2021-04-01 10:30 ` Phil Sutter @ 2021-04-01 11:44 ` Pablo Neira Ayuso 0 siblings, 0 replies; 6+ messages in thread From: Pablo Neira Ayuso @ 2021-04-01 11:44 UTC (permalink / raw) To: Phil Sutter, Florian Westphal, netfilter-devel On Thu, Apr 01, 2021 at 12:30:55PM +0200, Phil Sutter wrote: > Hi, > > On Wed, Mar 31, 2021 at 10:51:51PM +0200, Pablo Neira Ayuso wrote: > > On Wed, Mar 31, 2021 at 04:41:40PM +0200, Phil Sutter wrote: > > > On Wed, Mar 31, 2021 at 03:35:10PM +0200, Florian Westphal wrote: > > > > Phil Sutter <phil@nwl.cc> wrote: > > > > > I'm currently trying to fix for an issue in Kubernetes realm[1]: > > > > > Baseline is they are trying to restore a ruleset with ~700k lines and it > > > > > fails. Needless to say, legacy iptables handles it just fine. > > > > > > > > > > Meanwhile I found out there's a limit of 1024 iovecs when submitting the > > > > > batch to kernel, and this is what they're hitting. > > > > > > > > > > I can work around that limit by increasing each iovec (via > > > > > BATCH_PAGE_SIZE) but keeping pace with legacy seems ridiculous: > > > > > > > > > > With a scripted binary-search I checked the maximum working number of > > > > > restore items of: > > > > > > > > > > (1) User-defined chains > > > > > (2) rules with merely comment match present > > > > > (3) rules matching on saddr, daddr, iniface and outiface > > > > > > > > > > Here's legacy compared to nft with different factors in BATCH_PAGE_SIZE: > > > > > > > > > > legacy 32 (stock) 64 128 256 > > > > > ---------------------------------------------------------------------- > > > > > 1'636'799 1'602'202 - NC - - NC - - NC - > > > > > 1'220'159 302'079 604'160 1'208'320 - NC - > > > > > 3'532'040 242'688 485'376 971'776 1'944'576 > > > > > > > > Can you explain that table? What does 1'636'799 mean? NC? > > > > > > Ah, sorry: NC is "not care", I didn't consider those numbers relevant > > > given that iptables-nft has caught up to legacy previously already. > > > > > > 1'636'799 is the max number of user-defined chains I can successfully > > > restore using iptables-legacy-restore. Looks like I dropped the rows' > > > description while reformatting by accident: the first row of that table > > > corresponds with test (1), second with test (2) and third with test (3). > > > > > > So legacy may restore at once ~1.6M chains or ~1.2M comment rules or > > > ~3.5M rules with {s,d}{addr,iface} matches. > > > > > > The following columns are for iptables-nft with varying BATCH_PAGE_SIZE > > > values. Each of the (max 1024) iovecs passed to kernel via sendmsg() is > > > 'N * getpagesize()' large. > > > > Did you measure any slow down in the ruleset load time after selecting > > a larger batch chunk size? > > Restoring 100k rules shows no significant difference in between stock > (32 * 8k) and 512 * 8k chunk sizes. So if you think it's acceptable to > allocate 4MB of buffer at once, I'd just send a patch. That's fine. > Lifting that 1024 chunk count limit might be an alternative, but I guess > that sits in kernel space? That sits in the kernel, in the generic socket layer IIRC. P.S: Would you mind to send a patch for nftables too to keep it in sync? Thanks. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-04-01 17:58 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-03-31 9:13 iptables-nft fails to restore huge rulesets Phil Sutter 2021-03-31 13:35 ` Florian Westphal 2021-03-31 14:41 ` Phil Sutter 2021-03-31 20:51 ` Pablo Neira Ayuso 2021-04-01 10:30 ` Phil Sutter 2021-04-01 11:44 ` Pablo Neira Ayuso
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).