ct state vmap no longer works on 6.3 kernel

ct state vmap no longer works on 6.3 kernel

[Rvfg]

Hi. I recently upgraded to 6.3 kernel and I noticed my nftables starts 
dropping incoming ipv6 router advertisement packets. Here is my input 
chains:

|chain input {||
||        type filter hook input priority filter; policy drop;||
||        iifname "lo" accept comment "trusted interfaces"||
||        ct state vmap { invalid : drop, established : accept, related 
: accept, * : jump input-allow }||
||}||
||
||chain input-allow {||
||        meta l4proto ipv6-icmp meta nftrace set 1||
||        tcp dport { 22, 22000 } accept||
||        udp dport { 21027, 22000 } accept||
||        icmp type echo-request limit rate 20/second accept comment 
"allow ping"||
||        icmpv6 type != { nd-redirect, 139 } accept||
||        ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"||
||}|

This setup used to work fine. But now RA packets never go through the 
input-allow chain (not show up in nftrace). Something must be wrong in 
the "ct state vmap" rule. I'm able to work around this by adding a "jump 
input-allow" rule at the ending of "chain input".

I found 
https://github.com/torvalds/linux/commit/d9e7891476057b24a1acbf10a491e5b9a1c4ae77 
might be relevant (not tested yet). I'm not very familiar with the 
kernel. But the return NFT_BREAK in nft_ct_fast.c caught my eye. Is this 
the cause?

Thanks.

Re: ct state vmap no longer works on 6.3 kernel

[Pablo Neira Ayuso]

On Mon, May 01, 2023 at 09:58:52PM +0800, Rvfg wrote:
> Hi. I recently upgraded to 6.3 kernel and I noticed my nftables starts
> dropping incoming ipv6 router advertisement packets. Here is my input
> chains:
> 
> |chain input {||
> ||        type filter hook input priority filter; policy drop;||
> ||        iifname "lo" accept comment "trusted interfaces"||
> ||        ct state vmap { invalid : drop, established : accept, related :
> accept, * : jump input-allow }||
> ||}||
> ||
> ||chain input-allow {||
> ||        meta l4proto ipv6-icmp meta nftrace set 1||
> ||        tcp dport { 22, 22000 } accept||
> ||        udp dport { 21027, 22000 } accept||
> ||        icmp type echo-request limit rate 20/second accept comment "allow
> ping"||
> ||        icmpv6 type != { nd-redirect, 139 } accept||
> ||        ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"||
> ||}|
> 
> This setup used to work fine. But now RA packets never go through the
> input-allow chain (not show up in nftrace). Something must be wrong in the
> "ct state vmap" rule. I'm able to work around this by adding a "jump
> input-allow" rule at the ending of "chain input".
> 
> I found https://github.com/torvalds/linux/commit/d9e7891476057b24a1acbf10a491e5b9a1c4ae77
> might be relevant (not tested yet). I'm not very familiar with the kernel.
> But the return NFT_BREAK in nft_ct_fast.c caught my eye. Is this the cause?

I don't see anything bad with this patch.

Did you enable conntrack logging to understand why conntrack is
marking your packets as invalid?

# sh -c 'echo 58 > /proc/sys/net/netfilter/nf_conntrack_log_invalid'

where 58 is ICMPv6.

Re: ct state vmap no longer works on 6.3 kernel

[Rvfg]

On 5/3/23 14:19, Pablo Neira Ayuso wrote:
> I don't see anything bad with this patch.
> 
> Did you enable conntrack logging to understand why conntrack is
> marking your packets as invalid?
> 
> # sh -c 'echo 58 > /proc/sys/net/netfilter/nf_conntrack_log_invalid'
> 
> where 58 is ICMPv6.

Thanks for your reply. I tried enable conntrack logging. But nothing 
printed in dmesg. Actually on 6.2.13 kernel, those RA packets are 
untracked. Not invalid. Here's a trace on 6.2.13 kernel:

trace id d8e73ea3 inet nixos-fw input packet: iif "wlp1s0" ether saddr 
20:76:93:40:f4:a5 ether daddr a8:7e:ea:ed:dd:a2 ip6 saddr 
fe80::2276:93ff:fe40:f4a5 ip6 daddr fe80::aa7e:eaff:feed:dda2 ip6 dscp 
cs0 ip6 ecn not-ect ip6 hoplimit 255 ip6 flowlabel 804199 ip6 nexthdr 
ipv6-icmp ip6 length 120 icmpv6 type nd-router-advert icmpv6 code 
no-route icmpv6 parameter-problem 1086326536 @th,64,96 0x1012076
trace id d8e73ea3 inet nixos-fw input rule meta l4proto ipv6-icmp meta 
nftrace set 1 (verdict continue)
trace id d8e73ea3 inet nixos-fw input rule ct state untracked jump 
input-allow (verdict jump input-allow)
trace id d8e73ea3 inet nixos-fw input-allow rule icmpv6 type != { 
nd-redirect, 139 } accept

But on 6.3 kernel, none of the 5 ct states could match my packets. 
Here's another trace on 6.3 kernel:

trace id bc3d036f inet nixos-fw input packet: iif "wlp1s0" ether saddr 
20:76:93:40:f4:a5 ether daddr a8:7e:ea:ed:dd:a2 ip6 saddr 
fe80::2276:93ff:fe40:f4a5 ip6 daddr fe80::aa7e:eaff:feed:dda2 ip6 dscp 
cs0 ip6 ecn not-ect ip6 hoplimit 255 ip6 flowlabel 804199 ip6 nexthdr 
ipv6-icmp ip6 length 120 icmpv6 type nd-router-advert icmpv6 code 
no-route icmpv6 parameter-problem 1086326536 @th,64,96 0x1012076
trace id bc3d036f inet nixos-fw input rule meta l4proto ipv6-icmp meta 
nftrace set 1 (verdict continue)
trace id bc3d036f inet nixos-fw input verdict continue
trace id bc3d036f inet nixos-fw input policy drop

Re: ct state vmap no longer works on 6.3 kernel

[Florian Westphal]

Rvfg <i@rvf6.com> wrote:
> On 5/3/23 14:19, Pablo Neira Ayuso wrote:
> > I don't see anything bad with this patch.
> > 
> > Did you enable conntrack logging to understand why conntrack is
> > marking your packets as invalid?
> > 
> > # sh -c 'echo 58 > /proc/sys/net/netfilter/nf_conntrack_log_invalid'
> > 
> > where 58 is ICMPv6.
> 
> Thanks for your reply. I tried enable conntrack logging. But nothing printed
> in dmesg. Actually on 6.2.13 kernel, those RA packets are untracked. Not
> invalid. Here's a trace on 6.2.13 kernel:

Thanks, the BREAK in the referenced patch is the problem.

Please give this fix a try:

diff --git a/net/netfilter/nft_ct_fast.c b/net/netfilter/nft_ct_fast.c
--- a/net/netfilter/nft_ct_fast.c
+++ b/net/netfilter/nft_ct_fast.c
@@ -15,10 +15,6 @@ void nft_ct_get_fast_eval(const struct nft_expr *expr,
 	unsigned int state;
 
 	ct = nf_ct_get(pkt->skb, &ctinfo);
-	if (!ct) {
-		regs->verdict.code = NFT_BREAK;
-		return;
-	}
 
 	switch (priv->key) {
 	case NFT_CT_STATE:
@@ -30,6 +26,16 @@ void nft_ct_get_fast_eval(const struct nft_expr *expr,
 			state = NF_CT_STATE_INVALID_BIT;
 		*dest = state;
 		return;
+	default:
+		break;
+	}
+
+	if (!ct) {
+		regs->verdict.code = NFT_BREAK;
+		return;
+	}
+
+	switch (priv->key) {
 	case NFT_CT_DIRECTION:
 		nft_reg_store8(dest, CTINFO2DIR(ctinfo));
 		return;

Re: ct state vmap no longer works on 6.3 kernel

[Rvfg]

On 5/3/23 16:19, Florian Westphal wrote:
> Thanks, the BREAK in the referenced patch is the problem.
> 
> Please give this fix a try:
> 
> diff --git a/net/netfilter/nft_ct_fast.c b/net/netfilter/nft_ct_fast.c
> --- a/net/netfilter/nft_ct_fast.c
> +++ b/net/netfilter/nft_ct_fast.c
> @@ -15,10 +15,6 @@ void nft_ct_get_fast_eval(const struct nft_expr *expr,
>   	unsigned int state;
>   
>   	ct = nf_ct_get(pkt->skb, &ctinfo);
> -	if (!ct) {
> -		regs->verdict.code = NFT_BREAK;
> -		return;
> -	}
>   
>   	switch (priv->key) {
>   	case NFT_CT_STATE:
> @@ -30,6 +26,16 @@ void nft_ct_get_fast_eval(const struct nft_expr *expr,
>   			state = NF_CT_STATE_INVALID_BIT;
>   		*dest = state;
>   		return;
> +	default:
> +		break;
> +	}
> +
> +	if (!ct) {
> +		regs->verdict.code = NFT_BREAK;
> +		return;
> +	}
> +
> +	switch (priv->key) {
>   	case NFT_CT_DIRECTION:
>   		nft_reg_store8(dest, CTINFO2DIR(ctinfo));
>   		return;

This patch fixed my problem on 6.3.0 kernel. Thank you!