50k rules and performance issue in nft list table AND iptables-nft

50k rules and performance issue in nft list table AND iptables-nft

[Ricardo Katz]

Hello,

I've been digging into some performance issue I'm facing in my
production environment and would like to ask if someone has a light
about this.

My environment has ~50k rules that references some ipsets (it's a
Kubernetes cluster with Calico), and we've seen that sometimes
iptables-nft-save takes more than 20s. So I've tried to search what
was causing that, and have found some interesting behavior:

* iptables-nft-save performs better in newer kernels (5.4) than older
ones (4.19), but still the problem occurs
* nft list table performs WORST than iptables-nft-save, sometimes
taking more than 25s to display the rules. I've made the same test in
a non prod (less load) environment and it takes a little bit less but
yet, it's strange. The measured time is 4s in userspace and the rest
in kernel space, which leads me to ask: is there a way netlink should
be tuned? The production environment is not at its top load, but I'm
not going to focus in "nft" command right now.

So debugging iptables-nft-save I've seen that it walks through all the
rules to verify if they're compatible before printing them, something
like "4x nft_is_table_compatible (filter, raw, mangle, nat) -> X
nft_is_chain_compatible -> Y nft_is_rule_compatible -> Z
nft_is_expr_compatible that makes a string compare". This process
spends something like 5 to 7s in an average loaded environment.

After that, iptables-nft-save calls nft_rule_save which walks again
through the chains and rules, but this time to create them in the
format of iptables (nft_rule_print_save) spending some more kernel
time to get the content of the rule and print it.

I've seen a post
(https://developers.redhat.com/blog/2020/04/27/optimizing-iptables-nft-large-ruleset-performance-in-user-space/)
where there's some improvements into the userspace for the listing,
appending and deleting rule and being wondering: is this probably the
case also for iptables-nft-save, where it should have a cache and
iterate through it?

Also, if someone could provide some recommendations about checking if
this is a netlink pressure issue (this large time) or how can this be
improved. If further information is needed or anything I can help with
to improve this, please let me know.

Thank you!

Re: 50k rules and performance issue in nft list table AND iptables-nft

[Pablo Neira Ayuso]

Hi,

On Fri, Aug 14, 2020 at 06:56:32PM -0300, Ricardo Katz wrote:
> Hello,
> 
> I've been digging into some performance issue I'm facing in my
> production environment and would like to ask if someone has a light
> about this.
> 
> My environment has ~50k rules that references some ipsets (it's a
> Kubernetes cluster with Calico), and we've seen that sometimes
> iptables-nft-save takes more than 20s. So I've tried to search what
> was causing that, and have found some interesting behavior:
[...]
> * nft list table performs WORST than iptables-nft-save, sometimes
> taking more than 25s to display the rules. I've made the same test in
> a non prod (less load) environment and it takes a little bit less but
> yet, it's strange. The measured time is 4s in userspace and the rest
> in kernel space, which leads me to ask: is there a way netlink should
> be tuned? [...]

I have posted a patch to improve listing time:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/20200821111438.5362-2-pablo@netfilter.org/

Thanks for reporting.

Re: 50k rules and performance issue in nft list table AND iptables-nft

[Ricardo Katz]

Thank you Pablo.

I'll compile here with the patch ASAP and test in my QA and Prod
environment, and will provide some further feedback

Best,


On Fri, Aug 21, 2020 at 8:20 AM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> Hi,
>
> On Fri, Aug 14, 2020 at 06:56:32PM -0300, Ricardo Katz wrote:
> > Hello,
> >
> > I've been digging into some performance issue I'm facing in my
> > production environment and would like to ask if someone has a light
> > about this.
> >
> > My environment has ~50k rules that references some ipsets (it's a
> > Kubernetes cluster with Calico), and we've seen that sometimes
> > iptables-nft-save takes more than 20s. So I've tried to search what
> > was causing that, and have found some interesting behavior:
> [...]
> > * nft list table performs WORST than iptables-nft-save, sometimes
> > taking more than 25s to display the rules. I've made the same test in
> > a non prod (less load) environment and it takes a little bit less but
> > yet, it's strange. The measured time is 4s in userspace and the rest
> > in kernel space, which leads me to ask: is there a way netlink should
> > be tuned? [...]
>
> I have posted a patch to improve listing time:
>
> https://patchwork.ozlabs.org/project/netfilter-devel/patch/20200821111438.5362-2-pablo@netfilter.org/
>
> Thanks for reporting.