[Thread split] nftables rule optimization - dropping invalid in ingress?

[Thread split] nftables rule optimization - dropping invalid in ingress?

[William N.]

As per advice by Kerin Millar, this is a continuation of another
discussion [1] which resulted in a different topic.

On Sat, 20 Apr 2024 03:36:00 +0100 Kerin Millar wrote:

> To begin with, I would recommend that you jettison these rules
> outright. It is probable that they would otherwise end up being
> useless. But why? [...]

Actually, I have read about all this in older posts here. I should have
probably clarified better the forest, not just the trees.

The rules I mention (along with a few others) were inspired by a few
sources - some using iptables (where INVALID may be different in its
code definition from nftables and thus need such rules). That said, I
have actually tested and am aware that e.g. Xmas is an invalid TCP
packet that will be dropped by conntrack anyway. Similarly, the others
too.

However, in the setup I am trying to implement, I am attempting to be
"clever" and optimize things by dropping bad traffic earlier, so I am
doing it in the ingress hook where, AFAICS, conntrack is not available.
Why ingress? - Because I am following the general principle that
attacks should be stopped as soon and as far as possible, rather than
allowing them go further inside (in this case - next hooks). And even
though the next hook (prerouting) can drop e.g. Xmas of FINSYN as
invalid, I assume it would be a waste of CPU cycles to allow further
processing of such traffic. So, I thought: why not prevent the
unnecessary load on stateful conntrack? - Hence the whole idea to drop
early.

OTOH, adding more rules to ingress adds CPU cycles itself.

Which is more optimal - dropping early or not piling up extra rules in
ingress? Looking for an answer to that, I have done this:

As per earlier advise from you in a different context, I checked this:

# zgrep BPFILTER /proc/config.gz 
# CONFIG_BPFILTER is not set

If I am reading this correctly, it means there is no BPF JIT
optimization. Is this normal? Is BPF still experimental and for that
reason not available? I don't know, which is why I asked and still hope
for an answer:

https://marc.info/?l=netfilter&m=171345423924347&w=2

Why am I referring to BPF? - Because I suppose having it available
would make the difference between the "drop early" (in ingress) and
"drop as invalid" (in prerouting) cases negligible.

Now, the question comes down to: How big is the actual difference? Is
it negligible right now (without BPF)? - I really don't know. Hence
this other thread:

https://marc.info/?l=netfilter&m=171354240711565&w=2

Any info and advice is very welcome, as the whole thing discussed here
is very unclear to me.

--

[1] https://marc.info/?l=netfilter&m=171358042732609&w=2

Re: [Thread split] nftables rule optimization - dropping invalid in ingress?

[William N.]

After spending some time looking for more info and based on this:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/nf_conntrack_proto_tcp.c?h=v5.8#n700

I think I figured it out:

tcp flags & (fin|syn|rst|ack|urg) != {
		syn,
		syn|urg,
		syn|ack,
		rst,
		rst|ack,
		fin|ack,
		fin|ack|urg,
		ack,
		ack|urg
	} drop comment "TCP invalid"

This checks the listed values against the mask "fin|syn|rst|ack|urg".
The same values and mask are used in the conntrack code, i.e. it drops
invalid TCP packets.

According to my own tests, this works in the ingress hook, i.e. early
drop.

The only question that remains is performance measurement and
comparison, as mentioned.

Please let me know what you think.

Re: [Thread split] nftables rule optimization - dropping invalid in ingress?

[Kerin Millar]

On Sat, 20 Apr 2024, at 7:37 PM, William N. wrote:
> After spending some time looking for more info and based on this:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/nf_conntrack_proto_tcp.c?h=v5.8#n700
>
> I think I figured it out:
>
> tcp flags & (fin|syn|rst|ack|urg) != {
> 		syn,
> 		syn|urg,
> 		syn|ack,
> 		rst,
> 		rst|ack,
> 		fin|ack,
> 		fin|ack|urg,
> 		ack,
> 		ack|urg
> 	} drop comment "TCP invalid"
>
> This checks the listed values against the mask "fin|syn|rst|ack|urg".
> The same values and mask are used in the conntrack code, i.e. it drops
> invalid TCP packets.
>
> According to my own tests, this works in the ingress hook, i.e. early
> drop.
>
> The only question that remains is performance measurement and
> comparison, as mentioned.
>
> Please let me know what you think.

The rule looks good. Borrowing from the conntrack code was a bright idea.

If using the ingress hook in this way is to make any measurable difference to your load average at all, my expectation would be for it be observable in the event that you are subjected to a concentrated flood of invalid TCP packets. You could use hping3 to conduct a series of stress tests.

-- 
Kerin Millar

Re: [Thread split] nftables rule optimization - dropping invalid in ingress?

[William N.]

On Sat, 20 Apr 2024 20:16:49 +0100 Kerin Millar wrote:

> If using the ingress hook in this way is to make any measurable
> difference to your load average at all, my expectation would be for
> it be observable in the event that you are subjected to a
> concentrated flood of invalid TCP packets.

I was thinking the same.

> You could use hping3 to conduct a series of stress tests.

The question is how to measure the difference correctly. I am not a
network expert. I see some articles about firewall performance testing
using  iperf3 but in a VM-to-VM test (which I use) it is quite
inconsistent. Additionally, iperf3 is not able to do what hping3
does and I have no idea how to measure anything using hping3, which is
probably the correct test, as it would trigger the rules.

My clumsy attempt to have at least some comparison:

Testing procedure
-----------------

1. Design the ruleset
2. Put the rule to be tested in a separate file and include that file
3. In the file, have the line with the rule, followed by a single
'continue' on a new line. Repeat this combination 1000 times (to make
it have an observable effect)
4. In VM1 run (where rules reside):

iperf3 -s -p <port>
5. In VM2 run:

for (( i=0; i<10; i++)); do iperf3 -c 10.137.0.82 -p <port> -V; done

6. Note the speed after each iteration.

I am doing it 10 times because my observation shows that the first
iteration is always faster and the next ones decline to a point of
"saturation" (my interpretation - some resource exhaustion, e.g. memory
buffer). That is the inconsistency, mentioned above and my attempt to
get rid of it. Additionally, I also tested 'time ./my-firewall' to
compare this too.

Results
-------

# Early drop:

time to load = 23 sec
Bitrates (in Gbits/sec):

3.53
2.88
2.92
2.91
2.66
2.80
2.83
2.79
2.83
2.82

# Drop in prerouting (using conntrack and 'invalid'):

time to load = 1.326 s
Bitrates:

4.19
3.48
3.46
3.50
3.53
3.52
3.51
3.48
3.42
3.40

For comparison, the speed without any ruleset (after 'nft flush
ruleset'):

Bitrates:

5.54
4.96
4.66
4.33
4.32
4.32
4.33
4.34
4.31
4.32

Summarized
----------

Given the rules are repeated 1000 times, early drop is ~82% of the
speed of dropping using conntrack in prerouting and takes x20 more time
to load.

My interpretation (may be completely wrong, so corrections are welcome)
-----------------

conntrack is faster because it is compiled (binary), i.e. the same rule
written in nftables syntax needs additional processing (hence the much
higher load time), which makes it less efficient.

Additional info
---------------

If the rule is set only once (not repeated 1000 times, as in the test)
there is no difference between the 3 cases above - same speed. So,
ceteris paribus, the question comes down to: is there a security
benefit in dropping invalid packets earlier (e.g. right after dropping
fragments and bogons)?

I would highly appreciate the thoughts of the experts here.

Re: [Thread split] nftables rule optimization - dropping invalid in ingress?

[William N.]

Another test, using only invalid packets, shows the opposite result:

timeout 65s hping3 <host> --flood --xmas

When dropping in ingress:

14455889 packets transmitted
(on <host>) load average: 0.26, 0.30, 0.18

When dropping in prerouting (conntrack 'invalid'):

13794361 packets transmitted
(on <host>) load average: 0.55, 0.29, 0.19

So, after all, dropping in ingress seems faster (5% more packets for
the same time) and x2 less resource intensive.

Re: [Thread split] nftables rule optimization - dropping invalid in ingress?

[Eric]

Sent with Proton Mail secure email.

On Saturday, April 20th, 2024 at 11:37, William N. <netfilter@riseup.net> wrote:

> After spending some time looking for more info and based on this:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/nf_conntrack_proto_tcp.c?h=v5.8#n700
> 
> I think I figured it out:
> 
> tcp flags & (fin|syn|rst|ack|urg) != {
> syn,
> syn|urg,
> syn|ack,
> rst,
> rst|ack,
> fin|ack,
> fin|ack|urg,
> ack,
> ack|urg
> } drop comment "TCP invalid"
> 
> This checks the listed values against the mask "fin|syn|rst|ack|urg".
> The same values and mask are used in the conntrack code, i.e. it drops
> invalid TCP packets.
> 
> According to my own tests, this works in the ingress hook, i.e. early
> drop.
> 
> The only question that remains is performance measurement and
> comparison, as mentioned.
> 
> Please let me know what you think.

I'd be very interested in seeing some statistics on how many actual
invalid packets you see on a live link.  Stick some counters in there
and collect dropped versus passed packets...

My naive guess would be there are only tiny percentage of rejected
packets.

Eric

Re: [Thread split] nftables rule optimization - dropping invalid in ingress?

[Slavko]

Dňa 21. apríla 2024 3:45:31 UTC používateľ Eric <evil.function@proton.me> napísal:

>I'd be very interested in seeing some statistics on how many actual
>invalid packets you see on a live link.  Stick some counters in there
>and collect dropped versus passed packets...
>
>My naive guess would be there are only tiny percentage of rejected
>packets.

Your guess is not as naive, at least on my low traffic server i have ~70
drops (IPv4) in last 7 days. On my router it is ~370 drops in both, the
INPUT & FORWARD chains (iptables), in last ~100 days. That are
really tiny fractions of other drops, but many drops (by IP) happens
before INVALID rule.

regards


-- 
Slavko
https://www.slavino.sk/

Re: [Thread split] nftables rule optimization - dropping invalid in ingress?

[William N.]

On Sun, 21 Apr 2024 03:45:31 +0000 Eric wrote:

> I'd be very interested in seeing some statistics on how many actual
> invalid packets you see on a live link.  Stick some counters in there
> and collect dropped versus passed packets...

This particular system is a desktop one (rebooted often), so that kind
of stats won't make any sense.

> My naive guess would be there are only tiny percentage of rejected
> packets.

Without a particular attack - quite possible. However, it is always
good to learn what is better/worse/futile.