Proxy load balancer rules

Proxy load balancer rules

[Serguei Bezverkhi (sbezverk)]

Hello,

While running kubernetes e2e conformance tests against nfproxy, I found out that current kube-proxy builds kind of complicated set of rules, I was wondering if you could check to see if there is equivalents for keywords used in nftables:

If packet hits this loadbalancer ip, the processing starts:

-A KUBE-SERVICES -d 192.168.80.250/32 -p tcp -m comment --comment "services-9837/affinity-lb-esipp-transition: loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-BAJ42O6WMSSB7YGA

-A KUBE-FW-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition: loadbalancer IP" -j KUBE-XLB-BAJ42O6WMSSB7YGA
-A KUBE-FW-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition: loadbalancer IP" -j KUBE-MARK-DROP

-A KUBE-XLB-BAJ42O6WMSSB7YGA -s 57.112.0.0/12 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-BAJ42O6WMSSB7YGA
-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "masquerade LOCAL traffic for services-9837/affinity-lb-esipp-transition: LB IP" -m addrtype --src-type LOCAL -j KUBE-MARK-MASQ
-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "route LOCAL traffic for services-9837/affinity-lb-esipp-transition: LB IP to service chain" -m addrtype --src-type LOCAL -j KUBE-SVC-BAJ42O6WMSSB7YGA

!
!   -m recent --rcheck --seconds 10800 --reap  --rsource - keywords I am looking for equivalent in  nftables  
!

-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-JAOQ4ZBNFGZ34AZ4 --mask 255.255.255.255 --rsource -j KUBE-SEP-JAOQ4ZBNFGZ34AZ4
-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-WLHDVQTL57VBPURE --mask 255.255.255.255 --rsource -j KUBE-SEP-WLHDVQTL57VBPURE
-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-5XWCIKNI3M4MWAMU --mask 255.255.255.255 --rsource -j KUBE-SEP-5XWCIKNI3M4MWAMU
!
-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "Balancing rule 0 for services-9837/affinity-lb-esipp-transition:" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-JAOQ4ZBNFGZ34AZ4
-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "Balancing rule 1 for services-9837/affinity-lb-esipp-transition:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-WLHDVQTL57VBPURE
-A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "Balancing rule 2 for services-9837/affinity-lb-esipp-transition:" -j KUBE-SEP-5XWCIKNI3M4MWAMU


-A KUBE-SEP-5XWCIKNI3M4MWAMU -s 57.112.0.208/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-5XWCIKNI3M4MWAMU -p tcp -m recent --set --name KUBE-SEP-5XWCIKNI3M4MWAMU --mask 255.255.255.255 --rsource -m tcp -j DNAT [unsupported revision]

-A KUBE-SEP-JAOQ4ZBNFGZ34AZ4 -s 57.112.0.206/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-JAOQ4ZBNFGZ34AZ4 -p tcp -m recent --set --name KUBE-SEP-JAOQ4ZBNFGZ34AZ4 --mask 255.255.255.255 --rsource -m tcp -j DNAT [unsupported revision]

-A KUBE-SEP-WLHDVQTL57VBPURE -s 57.112.0.207/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-WLHDVQTL57VBPURE -p tcp -m recent --set --name KUBE-SEP-WLHDVQTL57VBPURE --mask 255.255.255.255 --rsource -m tcp -j DNAT [unsupported revision]

Appreciate a lot your help 
Thank you
Serguei

Re: Proxy load balancer rules

[Phil Sutter]

Hi Serguei,

On Thu, Jan 30, 2020 at 05:12:07PM +0000, Serguei Bezverkhi (sbezverk) wrote:
[...]
> 
> !
> !   -m recent --rcheck --seconds 10800 --reap  --rsource - keywords I am looking for equivalent in  nftables  
> !
> 
> -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-JAOQ4ZBNFGZ34AZ4 --mask 255.255.255.255 --rsource -j KUBE-SEP-JAOQ4ZBNFGZ34AZ4
> -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-WLHDVQTL57VBPURE --mask 255.255.255.255 --rsource -j KUBE-SEP-WLHDVQTL57VBPURE
> -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-5XWCIKNI3M4MWAMU --mask 255.255.255.255 --rsource -j KUBE-SEP-5XWCIKNI3M4MWAMU

There is no direct equivalent for recent extension in nftables (yet).
But in this case I think a set with timeout would do the trick.

The above simply checks if saddr is part of that set (--rcheck). The
value given in --seconds would be the set's default element timeout. No
need for --reap, elements will disappear automatically.

[...]
> -A KUBE-SEP-5XWCIKNI3M4MWAMU -s 57.112.0.208/32 -j KUBE-MARK-MASQ
> -A KUBE-SEP-5XWCIKNI3M4MWAMU -p tcp -m recent --set --name KUBE-SEP-5XWCIKNI3M4MWAMU --mask 255.255.255.255 --rsource -m tcp -j DNAT [unsupported revision]
> 
> -A KUBE-SEP-JAOQ4ZBNFGZ34AZ4 -s 57.112.0.206/32 -j KUBE-MARK-MASQ
> -A KUBE-SEP-JAOQ4ZBNFGZ34AZ4 -p tcp -m recent --set --name KUBE-SEP-JAOQ4ZBNFGZ34AZ4 --mask 255.255.255.255 --rsource -m tcp -j DNAT [unsupported revision]
> 
> -A KUBE-SEP-WLHDVQTL57VBPURE -s 57.112.0.207/32 -j KUBE-MARK-MASQ
> -A KUBE-SEP-WLHDVQTL57VBPURE -p tcp -m recent --set --name KUBE-SEP-WLHDVQTL57VBPURE --mask 255.255.255.255 --rsource -m tcp -j DNAT [unsupported revision]

These rules add saddr to the set or reset the timeout if already
present.

So, in order to replicate the above in nftables, you would:

* Add a new set for each different --name values given above
  - define a default timeout (suggested)
  - define a max size (suggested)
* Translate --rcheck into a simple set lookup
* Translate --set into set statement:
  'update @setxy { ip saddr timeout 10800 }'
  - use 'update' instead of 'add' to reset the timeout

For further info, please refer to nft manpage[1] as well as nftables
wiki[2].

Cheers, Phil

[1] 'SETS' and 'SET STATEMENT' sections in nft(8) 
[2] https://wiki.nftables.org/wiki-nftables/index.php/Updating_sets_from_the_packet_path

Re: Proxy load balancer rules

[Florian Westphal]

Phil Sutter <phil@nwl.cc> wrote:
> Hi Serguei,
> 
> On Thu, Jan 30, 2020 at 05:12:07PM +0000, Serguei Bezverkhi (sbezverk) wrote:
> [...]
> > 
> > !
> > !   -m recent --rcheck --seconds 10800 --reap  --rsource - keywords I am looking for equivalent in  nftables  
> > !
> > 
> > -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-JAOQ4ZBNFGZ34AZ4 --mask 255.255.255.255 --rsource -j KUBE-SEP-JAOQ4ZBNFGZ34AZ4
> > -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-WLHDVQTL57VBPURE --mask 255.255.255.255 --rsource -j KUBE-SEP-WLHDVQTL57VBPURE
> > -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-5XWCIKNI3M4MWAMU --mask 255.255.255.255 --rsource -j KUBE-SEP-5XWCIKNI3M4MWAMU
> 
> There is no direct equivalent for recent extension in nftables (yet).

Do we need one? All use cases I've seen can be handled via set infra.

Re: Proxy load balancer rules

[Phil Sutter]

Hi Florian,

On Fri, Jan 31, 2020 at 11:05:58PM +0100, Florian Westphal wrote:
> Phil Sutter <phil@nwl.cc> wrote:
> > Hi Serguei,
> > 
> > On Thu, Jan 30, 2020 at 05:12:07PM +0000, Serguei Bezverkhi (sbezverk) wrote:
> > [...]
> > > 
> > > !
> > > !   -m recent --rcheck --seconds 10800 --reap  --rsource - keywords I am looking for equivalent in  nftables  
> > > !
> > > 
> > > -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-JAOQ4ZBNFGZ34AZ4 --mask 255.255.255.255 --rsource -j KUBE-SEP-JAOQ4ZBNFGZ34AZ4
> > > -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-WLHDVQTL57VBPURE --mask 255.255.255.255 --rsource -j KUBE-SEP-WLHDVQTL57VBPURE
> > > -A KUBE-XLB-BAJ42O6WMSSB7YGA -m comment --comment "services-9837/affinity-lb-esipp-transition:" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-5XWCIKNI3M4MWAMU --mask 255.255.255.255 --rsource -j KUBE-SEP-5XWCIKNI3M4MWAMU
> > 
> > There is no direct equivalent for recent extension in nftables (yet).
> 
> Do we need one? All use cases I've seen can be handled via set infra.

Me neither, but in theory there are hard to achieve (--hitcount) or even missing
(--rttl) features. Support in iptables-translate would be interesting,
too, but that's a different kettle of fish. :)

Cheers, Phil