List chain during attack high CPU usage

List chain during attack high CPU usage

[Brskt]

Hello,

I'm using some filter which uses sets with meters. When the server is 
receiving a lot of packets per seconds (~800k+) and using the command to 
list the chain, it does take ~80% of CPU and take a long time to return 
an output.

# cat /etc/issue
Welcome to Alpine Linux 3.16

# nft -v
nftables v1.0.5 (Lester Gooch #4)

# uname -r
5.15.62-1-lts

Re: List chain during attack high CPU usage

[Pablo Neira Ayuso]

On Wed, Aug 31, 2022 at 02:18:17AM +0200, Brskt wrote:
> Hello,
> 
> I'm using some filter which uses sets with meters. When the server is
> receiving a lot of packets per seconds (~800k+) and using the command to
> list the chain, it does take ~80% of CPU and take a long time to return an
> output.
> 
> # cat /etc/issue
> Welcome to Alpine Linux 3.16
> 
> # nft -v
> nftables v1.0.5 (Lester Gooch #4)
> 
> # uname -r
> 5.15.62-1-lts

Could you provide more information on your setup? ruleset?

Ruleset listing is also lockless.

It is hard to guess with this short notice.

Re: List chain during attack high CPU usage

[Pablo Neira Ayuso]

On Thu, Sep 01, 2022 at 12:34:15PM +0200, Pablo Neira Ayuso wrote:
> On Wed, Aug 31, 2022 at 02:18:17AM +0200, Brskt wrote:
> > Hello,
> > 
> > I'm using some filter which uses sets with meters. When the server is
> > receiving a lot of packets per seconds (~800k+) and using the command to
> > list the chain, it does take ~80% of CPU and take a long time to return an
> > output.
> > 
> > # cat /etc/issue
> > Welcome to Alpine Linux 3.16
> > 
> > # nft -v
> > nftables v1.0.5 (Lester Gooch #4)
> > 
> > # uname -r
> > 5.15.62-1-lts
> 
> Could you provide more information on your setup? ruleset?
> 
> Ruleset listing is also lockless.
> 
> It is hard to guess with this short notice.

For the record, the existing userspace cache logic in nftables is
retrieving the elements in your set, and I suspect you have a lot of
elements, that is why:

        nft list chain x y

is likely taking time there to show the listing.

It should be possible to improve this cache logic, it will only
require a userspace software update.

Re: List chain during attack high CPU usage

[Brskt]

Hi,

Is there any updates on this ?

https://marc.info/?l=netfilter&m=166256224929259&w=2

I don't understand why using "nft list chain netdev firewall filter" 
take time and CPU usage even if a set have a high numbers of elements 
since we don't show the elements in the set.

There is the filter command in the chain "nft add rule netdev firewall 
filter update @ratelimit_test { ip saddr . ip daddr . th dport } counter 
drop" which use the set but, we don't see how many elements and/or which 
elements are in the set.

Listing a chain should not try to load the elements in the set(s) that 
are used in a filter like iptables with ipset does not.
It also do the same even if "counter" is not used.

Re: List chain during attack high CPU usage

[Pablo Neira Ayuso]

On Thu, Aug 03, 2023 at 11:49:59AM +0000, Brskt wrote:
> Hi,
> 
> Is there any updates on this ?
> 
> https://marc.info/?l=netfilter&m=166256224929259&w=2
> 
> I don't understand why using "nft list chain netdev firewall filter" take
> time and CPU usage even if a set have a high numbers of elements since we
> don't show the elements in the set.
> 
> There is the filter command in the chain "nft add rule netdev firewall
> filter update @ratelimit_test { ip saddr . ip daddr . th dport } counter
> drop" which use the set but, we don't see how many elements and/or which
> elements are in the set.
> 
> Listing a chain should not try to load the elements in the set(s) that are
> used in a filter like iptables with ipset does not.
> It also do the same even if "counter" is not used.

Patch to address this issue:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/20230822095324.23656-1-pablo@netfilter.org/