Change source or destination for packets arriving locally (for Direct Server Return)

Change source or destination for packets arriving locally (for Direct Server Return)

[Thomas Rosenstein]

Hello,

I'm trying to setup L3 load balancing (with direct server return) which 
requires me to send back or receive packets with a certain src/dst 
address,
but for these packets the dst address is replaced on the load balancer, 
then routed and are arriving on my linux container.

I tried with mangle, filter, nat tables and ip route, tc etc. and it 
seems nothing works as expected.

I tried to match via ip rule tos 0xc (2 additional bits) but that never 
matches, and ip rule doesn't allow arbitrary values
I tried to mark the packets in iptables mangle, and apply the ip rule on 
the fwmark, also never matches
I tried to match the fwmark with tc and use the nat action, also never 
matches (counters stay 0)
I tried to use PREROUTING with DNAT, but the packet is never there 
(packet to the local system)
The IP 10.253.1.18 is configured locally on eth0
The IP 10.253.253.163 is configured locally on eth0
The GW is 10.253.1.1
The IP 10.21.13.19 is in a completly different network segment

I'd like to do the following:

Incoming Packet: DSCP 0x3 Src: 10.21.13.19 Dst 10.253.1.18 SrcPort: 
45240 DstPort: 53

Match this packet based on DSCP = 0x3

Rewrite this Packet before it hits anything to:

DSCP: 0x0 Src: 10.21.13.19 Dst: 10.253.253.163 SrcPort: 45240 DstPort: 
53

Expected outcome:

The kernel sees this packet, and established the connection, all return 
traffic packets should look like this:

DSCP: 0x0 Src: 10.253.253.163 Dst 10.21.13.19 SrcPort: 53 DstPort: 45240

I do not want NAT (e.g. connection tracking, and reversing of the DNAT) 
- best would be a single rule that does this.
It looks like it can be done: 
https://www.slideshare.net/jschauma/l3dsr-overcoming-layer-2-limitations-of-direct-server-return-load-balancing

an illustration would be this: 
https://www.draw.io/?lightbox=1&highlight=0000ff&edit=_blank&layers=1&nav=1&title=Masq-Async-Routing.xml#R7Zxbc5s4FMc%2FjR%2BXQTcMj7m03Z3p7mQmndnmKSODbLPBiAElcfrpV4CwQSIJsYGQxp1pC0dCgH76Hx1d8AxdbLbfUpqs%2F%2BYBi2bQDrYzdDmD0LOJ%2FDc3PJUGAp3SsErDoDSBveE6%2FMWU0VbW%2BzBgWSOj4DwSYdI0%2BjyOmS8aNpqm%2FLGZbcmj5l0TumKG4dqnkWn9NwzEWlmBbe8T%2FmThaq1u7RKVsKD%2B3Srl97G63wyiZfGnTN7QqiyVP1vTgD%2FWTOjLDF2knIvyaLO9YFFetVW1ldd9fSZ199wpi0WXC7AC9UCje%2FXuWerP0Jm0AZdY0HYtgJAlC3YiWeL5IpVHq%2FwoyESZDwJoAddybWvHLxNPVVUKts1zr8UmkgYgD1OWhb%2Fooshgy%2FOEh7EogJHzGbmUFnoveFY2ivwCGoWrWB5HbJkX9cBSEUpWZ8oseCKtWUL9MF79yE8u%2F8DSsgyj6IJHPC0eBAWUuUs%2FzylSfsdqKY7vssVSppi1pyo0vyXb1kyqNr8xvmEifZJZVCqqWoJq%2BdjB5fljox2VtnWtCSFlo6rprnZF7%2FHJA0WwnaZr1D4LZLtWpzwVa77iMY2%2B7K3nsgUmeWoUxnc5nLztskChqVFj21D8rB3f5Fksos6uWBrKx2WpulBWYPr0s35Sy5%2Bf6hf8x4R4Uo4gxy9N%2B%2Bf9zguq4CVEGb9PffXWULkMmq5YVb2lKa%2BPFzGmLKIifGj6gWOYQENgBqQKwWa7yr2p5YeZzy1JQlZPpv5vwghott5B0tq5jRzXm7e1c%2BWJqpTKs8Gapr7TBYuupPhEyHNtLbgQfPOs6Cpl%2BpJG8Yw9KAjM3YaCADYVVGWpC4ig42F5IwhoL5qbWsrzAtpr5qaeNqiAkCkg%2FF4CQicBvU1A2Gl2QQCj0QSED4WVPYbCX7PMeuRpEUElt6XpE3HDTccHsW2GDjZoAddD6DAfxfN5ddc31eiBmM4PvpfzM8Pzjnry%2BSYp3N8tjYPbRNZXsmYpjTIroknZhj%2BJrIDuDU1VeS3xuNMDPUBGjyfsSUoKVIP4KQQU1cOcIorOg1q7qSHkeaNFFABNUENaTG557jgyahnYvp%2BMTiPbIwNz5LnjyWicAO8YGYFxNOSYGqqa8juIyOk%2FwEv8zyspUq08vCKpeQ9DJuC%2Bzo7FwVm%2BHpG%2FdkSzLNTYaDiWrs%2F81jnqhUswsV%2BqtFcbb61KSEvAW9k6t3F1h6t8%2Br4ebmvxtj6xXYpRXQVrSxNaQUbgrhdUKtgoqMC2e%2B1uJL3eSTIQENYqLM%2BZI%2Bp8AJLQI00AGB1GEtuvFNQfyWpdrU9NLpnTrslg7i3sj6BJos8IOt5hJPWCdlNUA5DsEF7WSMY8Zk2MmXwWYYIuzF%2FDaBe7xIE6a%2BkepYbdALexd%2BECOR9BxRhryPTZwq7s9ZVNo6Ae2XeY8z%2Bxf509ghoy51D2WkHIHY59hyWEV9g%2F56b7aBW73mDy7PXBpo6sc%2B%2BtO5CO7GVF06daNrXNo%2FMDV%2FfZN6WyxIMbVoe59DeGBr%2BBiyC6ssGBzcQoyBvORXQYNJ9IYgL7IYk9OBjJef8kj9jmNRGSeK4DONR16wXp%2FX%2BPJPufDPkdSLp9dcJ6QfZwJPufDPkdSKImAHCoJpE%2BrTVcP1k98onkS1Iydt4eqkk4nCYR7J3kcrmEz0xrOQuHfISIB8x70iQYUZPm1MblP2c%2FDJqT2TZ%2FzPRn93UcoqNE5p7FtpYE7ecbTecNpuZwYv8NhAct4LiWbcEXvoDQvpSYLMojVpfest%2BqibLykmN8AIHM8cT1pNV1zDJRdyQy6GiqyzaZDKYuc2CwU5f25dBH19dIMF3cgOmNqS9zbPB2mDIDQcXfKX8u9k79HjH3LwxGE5vjgxPNPj%2F%2BG1Wb2NzWOu3I8pg51ze4S%2BI0mKC5ueluqL4Pm8O2QwU2t8wR3ycjCTyNJDZJDqcuc1ly2pHlOJ87Y23kjD2z%2FxpMXea4LUweMhpsTDmZloylD2G%2BKbSlF2vLy9KsyKunTVKffajNbaoNE2CqDbeg7WNrJTbHcX9dTdgf9hEJYq9R3wSZSmr7TqkXJZljtGmHDsdMXr4hdLCbSHab6Mdwbi%2BMtLr%2BMEdj8upzo9Sc2W7H%2FxihAzGHWdeq85E3o3EQyQqSh2LNitf071i5T2YiwHoJE5pzFsQzlbSbzT9SSvJ0%2F%2Fs55Qz9%2FjeK0Jf%2FAQ%3D%3D

It's just missing the DSCP part.

BR
Thomas Rosenstein

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Arturo Borrero Gonzalez]

On 12 September 2017 at 08:00, Thomas Rosenstein
<thomas.rosenstein@creamfinance.com> wrote:
> Hello,
>
> I'm trying to setup L3 load balancing (with direct server return) which
> requires me to send back or receive packets with a certain src/dst address,
> but for these packets the dst address is replaced on the load balancer, then
> routed and are arriving on my linux container.
>


I guess you could do this with nftables. You can perform this kind of
load balancing with nftables out of the box [0].
Note that nftables should be able to work with DSCP, so you can
combine both things (matching, load-balancing) with the same
technology.

Please, read the docs in our wiki and do some tests.  After that, it
would be great if you come back here and report your experience :-)
Perhaps we can generate a concrete example and put it in the wiki for
future references.

[0] https://wiki.nftables.org/wiki-nftables/index.php/Load_balancing

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Thomas Rosenstein]

Hi,

I have to check it out, but in the mean time I already wrote my small 
iptables plugin to rewrite the dst-addr.

let's call it pre-alpha:

https://github.com/creamfinance/dstwrite

BR
Thomas

On 13 Sep 2017, at 11:34, Arturo Borrero Gonzalez wrote:

> On 12 September 2017 at 08:00, Thomas Rosenstein
> <thomas.rosenstein@creamfinance.com> wrote:
>> Hello,
>>
>> I'm trying to setup L3 load balancing (with direct server return) 
>> which
>> requires me to send back or receive packets with a certain src/dst 
>> address,
>> but for these packets the dst address is replaced on the load 
>> balancer, then
>> routed and are arriving on my linux container.
>>
>
>
> I guess you could do this with nftables. You can perform this kind of
> load balancing with nftables out of the box [0].
> Note that nftables should be able to work with DSCP, so you can
> combine both things (matching, load-balancing) with the same
> technology.
>
> Please, read the docs in our wiki and do some tests.  After that, it
> would be great if you come back here and report your experience :-)
> Perhaps we can generate a concrete example and put it in the wiki for
> future references.
>
> [0] https://wiki.nftables.org/wiki-nftables/index.php/Load_balancing

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Pablo Neira Ayuso]

On Wed, Sep 13, 2017 at 11:36:56AM +0200, Thomas Rosenstein wrote:
> Hi,
> 
> I have to check it out, but in the mean time I already wrote my small
> iptables plugin to rewrite the dst-addr.
> 
> let's call it pre-alpha:
> 
> https://github.com/creamfinance/dstwrite

Did you try packet field mangling?

https://wiki.nftables.org/wiki-nftables/index.php/Mangle_packet_header_fields

You need a Linux kernel >= 4.10.

Syntax is simple, eg.

        ip daddr set 1.2.3.4

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Thomas Rosenstein]

Hi,

it is now working and my router receives the return packets, but they 
are not forwarded and dropped after the MANGLE PREROUTING chain.

What could be the issue, or where in the kernel could I check what's 
happening? Is there some flag to enable detailed tracing?

Based on tcpdump the packet is okay regarding checksums.
Based on this 
https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg 
it breaks somwhere between mangle and nat PREROUTING.

Any suggestions or ideas are welcome!
Thanks

BR
Thomas

On 13 Sep 2017, at 11:34, Arturo Borrero Gonzalez wrote:

> On 12 September 2017 at 08:00, Thomas Rosenstein
> <thomas.rosenstein@creamfinance.com> wrote:
>> Hello,
>>
>> I'm trying to setup L3 load balancing (with direct server return) 
>> which
>> requires me to send back or receive packets with a certain src/dst 
>> address,
>> but for these packets the dst address is replaced on the load 
>> balancer, then
>> routed and are arriving on my linux container.
>>
>
>
> I guess you could do this with nftables. You can perform this kind of
> load balancing with nftables out of the box [0].
> Note that nftables should be able to work with DSCP, so you can
> combine both things (matching, load-balancing) with the same
> technology.
>
> Please, read the docs in our wiki and do some tests.  After that, it
> would be great if you come back here and report your experience :-)
> Perhaps we can generate a concrete example and put it in the wiki for
> future references.
>
> [0] https://wiki.nftables.org/wiki-nftables/index.php/Load_balancing

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Arturo Borrero Gonzalez]

On 13 September 2017 at 15:23, Thomas Rosenstein
<thomas.rosenstein@creamfinance.com> wrote:
> Hi,
>
> it is now working and my router receives the return packets, but they are
> not forwarded and dropped after the MANGLE PREROUTING chain.
>
> What could be the issue, or where in the kernel could I check what's
> happening? Is there some flag to enable detailed tracing?
>
> Based on tcpdump the packet is okay regarding checksums.
> Based on this
> https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg
> it breaks somwhere between mangle and nat PREROUTING.
>
> Any suggestions or ideas are welcome!

Please, avoid top-posting, thanks :-)

There is information about debugging in the wiki [0], and is rather easy to use.

[0] https://wiki.nftables.org/wiki-nftables/index.php/Ruleset_debug/tracing

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Thomas Rosenstein]

>
> Please, avoid top-posting, thanks :-)
>
> There is information about debugging in the wiki [0], and is rather 
> easy to use.
>
> [0] 
> https://wiki.nftables.org/wiki-nftables/index.php/Ruleset_debug/tracing

Okay :)

I added all the debug logs I can think of:

table ip filter {
	chain filter-prerouting-0 {
		type filter hook prerouting priority 0; policy accept;
		ip protocol icmp nftrace set 1
	}

	chain test2 {
	    type filter hook forward priority 0; policy accept;
	    ip protocol icmp nftrace set 1
	}

	chain test-200 {
	    type filter hook prerouting priority 200; policy accept;
	    ip protocol icmp nftrace set 1
	}

	chain test-m500 {
	    type nat hook prerouting priority -500; policy accept;
	    ip protocol icmp nftrace set 1
	}

	chain test-500 {
	    type filter hook prerouting priority 500; policy accept;
	    ip protocol icmp nftrace set 1
	}

	chain forward-0 {
	    type filter hook forward priority 0; policy accept;
	    ip protocol icmp nftrace set 1
	}

	chain forward-m500 {
     		type filter hook forward priority -500; policy accept;
	    ip protocol icmp nftrace set 1
	}
}

and the packets go missing between test-500 and forward-m500 also the 
test-m500 chain is never hit (nat).

Sorry for the names, I tried to rename them with:

nft rename chain ip filter test filter-prerouting-0

<cmdline>:1:1-47: Error: Could not process rule: No such file or 
directory
rename chain ip filter test filter-prerouting-0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

nft monitor logs:

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
daddr 18:a9:05:79:73:6e ip saddr 10.21.13.5 ip daddr 10.253.10.12 ip 
dscp cs0 ip ecn not-ect ip ttl 63 ip id 24831 ip length 84 icmp type 
echo-request icmp code 0 icmp id 21685 icmp sequence 1
trace id 42c618bd ip filter test2 rule ip protocol icmp nftrace set 1 
(verdict continue)
trace id 42c618bd ip filter test2 verdict continue
trace id 42c618bd ip filter test2
trace id 42c618bd ip filter forward-0 packet: iif eth0 oif kube-bridge 
ether saddr 9c:8e:99:0f:7e:46 ether daddr 18:a9:05:79:73:6e ip saddr 
10.21.13.5 ip daddr 10.253.10.12 ip dscp cs0 ip ecn not-ect ip ttl 63 ip 
id 24831 ip length 84 icmp type echo-request icmp code 0 icmp id 21685 
icmp sequence 1
trace id 42c618bd ip filter forward-0 rule ip protocol icmp nftrace set 
1 (verdict continue)
trace id 42c618bd ip filter forward-0 verdict continue
trace id 42c618bd ip filter forward-0
trace id a1f8a919 ip filter test packet: iif kube-bridge ether saddr 
0a:58:0a:fd:0a:0c ether daddr 0a:58:0a:fd:0a:01 ip saddr 10.253.254.236 
ip daddr 10.21.13.5 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 18916 ip 
length 84 icmp type echo-reply icmp code 0 icmp id 21685 icmp sequence 1
trace id a1f8a919 ip filter test rule ip protocol icmp nftrace set 1 
(verdict continue)
trace id a1f8a919 ip filter test verdict continue
trace id a1f8a919 ip filter test
trace id a1f8a919 ip filter test-200 packet: iif kube-bridge ether saddr 
0a:58:0a:fd:0a:0c ether daddr 0a:58:0a:fd:0a:01 ip saddr 10.253.254.236 
ip daddr 10.21.13.5 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 18916 ip 
length 84 icmp type echo-reply icmp code 0 icmp id 21685 icmp sequence 1
trace id a1f8a919 ip filter test-200 rule ip protocol icmp nftrace set 1 
(verdict continue)
trace id a1f8a919 ip filter test-200 verdict continue
trace id a1f8a919 ip filter test-200
trace id a1f8a919 ip filter test-500 packet: iif kube-bridge ether saddr 
0a:58:0a:fd:0a:0c ether daddr 0a:58:0a:fd:0a:01 ip saddr 10.253.254.236 
ip daddr 10.21.13.5 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 18916 ip 
length 84 icmp type echo-reply icmp code 0 icmp id 21685 icmp sequence 1
trace id a1f8a919 ip filter test-500 rule ip protocol icmp nftrace set 1 
(verdict continue)
trace id a1f8a919 ip filter test-500 verdict continue
trace id a1f8a919 ip filter test-500
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^


any idea where I could poke, what I could start to figure out why the 
packets are dropped in the kernel?

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Arturo Borrero Gonzalez]

On 13 September 2017 at 16:14, Thomas Rosenstein
<thomas.rosenstein@creamfinance.com> wrote:
>>
>> Please, avoid top-posting, thanks :-)
>>
>> There is information about debugging in the wiki [0], and is rather easy
>> to use.
>>
>> [0]
>> https://wiki.nftables.org/wiki-nftables/index.php/Ruleset_debug/tracing
>
>
> Okay :)
>
> I added all the debug logs I can think of:
>
> table ip filter {
>         chain filter-prerouting-0 {
>                 type filter hook prerouting priority 0; policy accept;
>                 ip protocol icmp nftrace set 1
>         }
>
>         chain test2 {
>             type filter hook forward priority 0; policy accept;
>             ip protocol icmp nftrace set 1
>         }
>
>         chain test-200 {
>             type filter hook prerouting priority 200; policy accept;
>             ip protocol icmp nftrace set 1
>         }
>
>         chain test-m500 {
>             type nat hook prerouting priority -500; policy accept;
>             ip protocol icmp nftrace set 1
>         }
>
>         chain test-500 {
>             type filter hook prerouting priority 500; policy accept;
>             ip protocol icmp nftrace set 1
>         }
>
>         chain forward-0 {
>             type filter hook forward priority 0; policy accept;
>             ip protocol icmp nftrace set 1
>         }
>
>         chain forward-m500 {
>                 type filter hook forward priority -500; policy accept;
>             ip protocol icmp nftrace set 1
>         }
> }
>
> and the packets go missing between test-500 and forward-m500 also the
> test-m500 chain is never hit (nat).
>

beware that NAT-type chains only see first packet of flows [0].

In the ruleset I see a rather complex mixture of hooks/priorities.
I would recommend to not use such a complex scheme unless you really
know what you are doing.
Better to simply configure a ruleset scheme similar to what is
included in iptables, using same hooks/priorities as described in the
docs [0].

We follow a different approach in nftables than in iptables. In
iptables, all comes pre-configured, which is easier
but may force you to have things you don't use. In nftables all is
open-configured, which could be a bit more complex
if you don't try to simplify.

[0] https://wiki.nftables.org/wiki-nftables/index.php/Configuring_chains

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Thomas Rosenstein]

>
> beware that NAT-type chains only see first packet of flows [0].
>
> In the ruleset I see a rather complex mixture of hooks/priorities.
> I would recommend to not use such a complex scheme unless you really
> know what you are doing.
> Better to simply configure a ruleset scheme similar to what is
> included in iptables, using same hooks/priorities as described in the
> docs [0].
>
> We follow a different approach in nftables than in iptables. In
> iptables, all comes pre-configured, which is easier
> but may force you to have things you don't use. In nftables all is
> open-configured, which could be a bit more complex
> if you don't try to simplify.
>
> [0] 
> https://wiki.nftables.org/wiki-nftables/index.php/Configuring_chains

Arturo, I'm aware of this, and I only used this complex scheme to find 
out WHY and WHERE my packets are getting lost.
With the structure I chose I can be sure that no other hook is dropping 
them.

Since neither iptables, nor nftables give any hint why they may be 
DROPPED by the kernel at the routing decision I have to rely on 
information from the community.
So where in the kernel does that routing decision after the PREROUTING 
hooks happen?

rp_filters are disabled by the way.

Re: Change source or destination for packets arriving locally (for Direct Server Return)

[Thomas Rosenstein]

>
> beware that NAT-type chains only see first packet of flows [0].
>
> In the ruleset I see a rather complex mixture of hooks/priorities.
> I would recommend to not use such a complex scheme unless you really
> know what you are doing.
> Better to simply configure a ruleset scheme similar to what is
> included in iptables, using same hooks/priorities as described in the
> docs [0].
>
> We follow a different approach in nftables than in iptables. In
> iptables, all comes pre-configured, which is easier
> but may force you to have things you don't use. In nftables all is
> open-configured, which could be a bit more complex
> if you don't try to simplify.
>
> [0] 
> https://wiki.nftables.org/wiki-nftables/index.php/Configuring_chains


Actually, I figured it out just now, the src ip of the packet is defined 
on one of the interfaces.

That's why the kernel is dropping the packet.

Is there some flag to stop it from doing that?