Ipv6 address in concatenation

Ipv6 address in concatenation

[Serguei Bezverkhi (sbezverk)]

Hello,

I started testing  nfproxy in ipv6 enabled kubernetes cluster and it seems ipv6 address cannot be a part of concatenation expression. Is there a known issue or it is me doing something incorrect?
From my side the code is the same, I just change ip4_addr to ip6_addr when I build sets.

        map no-endpoints {
                type inet_proto . ipv6_addr . inet_service : verdict
        }

        map do-mark-masq {
                type inet_proto . ipv6_addr . inet_service : verdict
        }

        map cluster-ip {
                type inet_proto . ipv6_addr . inet_service : verdict
        }

Thank you
Serguei

Re: Ipv6 address in concatenation

[Phil Sutter]

Hi,

On Thu, Feb 27, 2020 at 04:21:40PM +0000, Serguei Bezverkhi (sbezverk) wrote:
> I started testing  nfproxy in ipv6 enabled kubernetes cluster and it seems ipv6 address cannot be a part of concatenation expression. Is there a known issue or it is me doing something incorrect?
> From my side the code is the same, I just change ip4_addr to ip6_addr when I build sets.
> 
>         map no-endpoints {
>                 type inet_proto . ipv6_addr . inet_service : verdict
>         }
> 
>         map do-mark-masq {
>                 type inet_proto . ipv6_addr . inet_service : verdict
>         }
> 
>         map cluster-ip {
>                 type inet_proto . ipv6_addr . inet_service : verdict
>         }

Works fine for me. Wrong table family?

Cheers, Phil

Re: Ipv6 address in concatenation

[Florian Westphal]

Serguei Bezverkhi (sbezverk) <sbezverk@cisco.com> wrote:
> Hello,
> 
> I started testing  nfproxy in ipv6 enabled kubernetes cluster and it seems ipv6 address cannot be a part of concatenation expression. Is there a known issue or it is me doing something incorrect?
> From my side the code is the same, I just change ip4_addr to ip6_addr when I build sets.

types are irrelvant for the kernel.  They are ONLY used by the nft tool
so it knows how to format output.

I suspect you need to fix up the generated payload expressions
for ipv6.  Essentially, in the ipv6 case, you have a concatenation

ipv4_addr . ipv4_addr . ip4_addr . ipv4_addr . inet_service

(ipv6 address needs 4 32-bit registers)

i.e., you need to use a different destination register when you store
the tcp/udp port, else you will clobber a part of the ipv6 address.

Re: Ipv6 address in concatenation

[Florian Westphal]

Florian Westphal <fw@strlen.de> wrote:
> Serguei Bezverkhi (sbezverk) <sbezverk@cisco.com> wrote:
> > Hello,
> > 
> > I started testing  nfproxy in ipv6 enabled kubernetes cluster and it seems ipv6 address cannot be a part of concatenation expression. Is there a known issue or it is me doing something incorrect?
> > From my side the code is the same, I just change ip4_addr to ip6_addr when I build sets.
> 
> types are irrelvant for the kernel.  They are ONLY used by the nft tool
> so it knows how to format output.
> 
> I suspect you need to fix up the generated payload expressions
> for ipv6.  Essentially, in the ipv6 case, you have a concatenation
> 
> ipv4_addr . ipv4_addr . ip4_addr . ipv4_addr . inet_service
> 
> (ipv6 address needs 4 32-bit registers)
> 
> i.e., you need to use a different destination register when you store
> the tcp/udp port, else you will clobber a part of the ipv6 address.

Addendum: you can check with

nft --debug=netlink list ruleset

to make sure the registers get populated as expected by lookup
expression.

Re: Ipv6 address in concatenation

[Serguei Bezverkhi (sbezverk)]

Ok, I figured out the map issue, it was a length of the key in bits, damn copy/paste (

Appreciate if somebody could comment about the following:

sudo nft --debug=netlink insert rule ip6 kube-nfproxy-v6 k8s-nat-services ip6 nexthdr . ip6 daddr . th dport vmap @cluster-ip-set
ip6 kube-nfproxy-v6 k8s-nat-services 
  [ payload load 1b @ network header + 6 => reg 1 ]
  [ payload load 16b @ network header + 24 => reg 9 ].       < -- Is it loading reg 9 4-bytes, reg 10 4 bytes etc until reg 12? Or because the data 16 bytes long it has to skip 3 more register?
  [ payload load 2b @ transport header + 2 => reg 13 ]
  [ lookup reg 1 set cluster-ip-set dreg 0 ]

I am just trying to figure out how to calculate next register to use.    If there is algorithm for both ipv4 and ipv6 that would be awesome to know.

Thank you
Serguei

On 2020-02-27, 2:42 PM, "Florian Westphal" <fw@strlen.de> wrote:

    Serguei Bezverkhi (sbezverk) <sbezverk@cisco.com> wrote:
    > Hello,
    > 
    > I started testing  nfproxy in ipv6 enabled kubernetes cluster and it seems ipv6 address cannot be a part of concatenation expression. Is there a known issue or it is me doing something incorrect?
    > From my side the code is the same, I just change ip4_addr to ip6_addr when I build sets.
    
    types are irrelvant for the kernel.  They are ONLY used by the nft tool
    so it knows how to format output.
    
    I suspect you need to fix up the generated payload expressions
    for ipv6.  Essentially, in the ipv6 case, you have a concatenation
    
    ipv4_addr . ipv4_addr . ip4_addr . ipv4_addr . inet_service
    
    (ipv6 address needs 4 32-bit registers)
    
    i.e., you need to use a different destination register when you store
    the tcp/udp port, else you will clobber a part of the ipv6 address.
    
    

Re: Ipv6 address in concatenation

[Florian Westphal]

Serguei Bezverkhi (sbezverk) <sbezverk@cisco.com> wrote:
> Ok, I figured out the map issue, it was a length of the key in bits, damn copy/paste (
> 
> Appreciate if somebody could comment about the following:
> 
> sudo nft --debug=netlink insert rule ip6 kube-nfproxy-v6 k8s-nat-services ip6 nexthdr . ip6 daddr . th dport vmap @cluster-ip-set
> ip6 kube-nfproxy-v6 k8s-nat-services 
>   [ payload load 1b @ network header + 6 => reg 1 ]
>   [ payload load 16b @ network header + 24 => reg 9 ].       < -- Is it loading reg 9 4-bytes, reg 10 4 bytes etc until reg 12? Or because the data 16 bytes long it has to skip 3 more register?
>   [ payload load 2b @ transport header + 2 => reg 13 ]

The 'registers' are adjacent in memory, so loading 16 bytes to reg9
will also store data to 10, 11 and 12.

In case there are not enough next registers kernel will reject the
transaction.

>   [ lookup reg 1 set cluster-ip-set dreg 0 ]
> 
> I am just trying to figure out how to calculate next register to use.    If there is algorithm for both ipv4 and ipv6 that would be awesome to know.

See netlink_gen_concat() in src/netlink_linearize.c in nftables.

Its basically enough to take the start register and then add the
length, rounded up to 4 (register is always 4 byte).

So for the above you need 1 + 1 * 4 + 1, i.e. 6 registers,
then pass the first/start register to the lookup expression.

The lookup expression takes the number of 'next registers' it
needs to look at from the set key length.