nftables DNAT routes to wrong iface

nftables DNAT routes to wrong iface

[Marc SCHAEFER]

Hello,

I am trying to do this:

Both local IP addresses, when access from outside:

   enp2s0.310: 193.72.186.130:8080
   br0: 46.140.72.218:8080

are DNATed to

   enp2s0.202: 192.168.202.10:80 (a remote machine)

I know there are two steps:

   - the incoming DNAT in prerouting (and the forward accept)
     -> this part I am stuck, see below

   - then when the reply comes back, route to the proper interface where
     it came from (using conntrack + marks + specific routing tables)
     -> this part I have not done yet -- it would be required for
        193.72.186.130:8080 obviously because the default route
        does not go there.

What I observe:

   telnet 46.140.72.218 8080 from outside works (*), it connects to 192.168.202.10:80
   and there is nothing bizarre in tcpdump either on br0 nor on
   enp2s0.202 (no delays, lost packets, e.g.) (**)

   telnet 193.72.186.130 8080 gives this on enp2s0.300

      IP 193.72.186.190.52636 > 193.72.186.130.8080 (normal)
      IP 193.72.186.190.52636 > 192.168.202.10.80   (good, it was DNATted, BUT should be on enp2s0.202!)

   aka the DNAT is executed, but then 192.168.202.10 is not routed
   correctly.  From the diagram https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks
   I thought that routing would be done AFTER prerouting/DNAT.

The routing table has:

default via 46.140.72.217 dev br0 onlink 
46.140.72.216/29 dev br0 proto kernel scope link src 46.140.72.218 
192.168.202.0/24 dev enp2s0.202 proto kernel scope link src 192.168.202.2 
193.72.186.128/26 dev enp2s0.300 proto kernel scope link src 193.72.186.130 

(*) not from 193.72.186.0/24, however, gets the same bug.

The nftables config:

table ip filter {
        # obviously a later goal is also to encode the L4 protocol here
        # and not hardcode it in the prerouting
        map multihoming_ext {
                type ipv4_addr . inet_service : ipv4_addr . inet_service
                elements = { 193.72.186.130 . 8080 : 192.168.202.10 . 80,
                             46.140.72.218 . 8080 : 192.168.202.10 . 80 }
        }

        set w_all {
                type ipv4_addr
                flags interval
                elements = { 46.140.72.216/29,
                             192.168.202.1, 193.72.186.0/24
                           }
        }

        chain input {
                type filter hook input priority filter; policy drop;
                ct state invalid counter packets 3 bytes 120 drop
                ct state { established, related } counter packets 1092 bytes 76846 accept
                iif "lo" counter packets 0 bytes 0 accept
                counter packets 324 bytes 13539 jump whitelist
                counter packets 323 bytes 13455 jump blacknets
                counter packets 323 bytes 13455 jump blacklist
                counter packets 323 bytes 13455 jump incoming
        }

        chain forward {
                type filter hook forward priority filter; policy drop;
                ct state invalid counter packets 0 bytes 0 drop
                ct state { established, related } counter packets 12 bytes 548 accept

                # already after DNAT, obviously
                iifname "br0" ip daddr 192.168.202.10 tcp dport 80 accept
                iifname "enp2s0.300" ip daddr 192.168.202.10 tcp dport 80 accept
        }

        chain output {
                type filter hook output priority filter; policy accept;
                ct state invalid counter packets 0 bytes 0 drop
                ct state { established, related } counter packets 890 bytes 182096 accept
                oif "lo" counter packets 0 bytes 0 accept
                counter packets 1 bytes 76 jump outgoing
        }

        chain rejectcounter {
                meta l4proto tcp counter packets 0 bytes 0 reject with tcp reset
                meta l4proto udp counter packets 0 bytes 0 reject
                counter packets 0 bytes 0 drop
        }

        chain dropcounter {
                counter packets 320 bytes 13349 drop
        }

        chain whitelist {
                ip saddr @w_all counter packets 1 bytes 84 accept
        }

        chain blacknets {
        }

        chain blacklist {
        }

        chain incoming {
                icmp type echo-request counter packets 3 bytes 106 accept
                icmp type echo-reply counter packets 0 bytes 0 accept
                tcp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
                udp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
                counter packets 320 bytes 13349 jump dropcounter
        }

        chain outgoing {
        }

        chain multihoming_prerouting {
                type nat hook prerouting priority dstnat; policy accept;

                # this is the DNAT
                dnat ip to ip daddr . tcp dport map @multihoming_ext
        }
}
table ip myhelpers {
        chain prerouting {
                type filter hook prerouting priority filter; policy accept;
        }
}

[ I removed ip6 entries ]

Any idea what could be wrong?

Thank you.

(**) very nice:
br0:
IP 46.140.72.222.60394 > 46.140.72.218.8080
IP 46.140.72.218.8080 > 46.140.72.222.60394
IP 46.140.72.222.60394 > 46.140.72.218.8080

enp2s0.202 (another connection):
IP 46.140.72.222.57790 > 192.168.202.10.80
IP 192.168.202.10.80 > 46.140.72.222.57790
IP 46.140.72.222.57790 > 192.168.202.10.80

Re: nftables DNAT routes to wrong iface

[Sunny73Cr]

Hi,

> The routing table has:

> default via 46.140.72.217 dev br0 onlink
> 46.140.72.216/29 dev br0 proto kernel scope link src 46.140.72.218
> 192.168.202.0/24 dev enp2s0.202 proto kernel scope link src > 192.168.202.2
> 193.72.186.128/26 dev enp2s0.300 proto kernel scope link src 193.72.186.130

Oops! Old reply (please disregard): '193.72.186.128/26' is poorly subnetted; the correct base address is 193.72.186.192/26. Did you mean '193.72.186.128/25'? As a result, '193.72.186.130' is not part of the subnet that you are routing. Consider if 25 bits is the correct mask.

New reply (and now public): '193.72.186.128/26' does not cover all of '193.72.186.0/24', and some packets will not get routed. You could try changing your route to '193.72.186.0/24', or edit the firewall policy to reflect the '193.72.186.128/26' route.

sunny

Sent with Proton Mail secure email.

On Sunday, January 26th, 2025 at 4:55 AM, Marc SCHAEFER <schaefer@alphanet.ch> wrote:

> Hello,
> 
> I am trying to do this:
> 
> Both local IP addresses, when access from outside:
> 
> enp2s0.310: 193.72.186.130:8080
> br0: 46.140.72.218:8080
> 
> are DNATed to
> 
> enp2s0.202: 192.168.202.10:80 (a remote machine)
> 
> I know there are two steps:
> 
> - the incoming DNAT in prerouting (and the forward accept)
> -> this part I am stuck, see below
> 
> 
> - then when the reply comes back, route to the proper interface where
> it came from (using conntrack + marks + specific routing tables)
> -> this part I have not done yet -- it would be required for
> 
> 193.72.186.130:8080 obviously because the default route
> does not go there.
> 
> What I observe:
> 
> telnet 46.140.72.218 8080 from outside works (*), it connects to 192.168.202.10:80
> and there is nothing bizarre in tcpdump either on br0 nor on
> enp2s0.202 (no delays, lost packets, e.g.) (**)
> 
> telnet 193.72.186.130 8080 gives this on enp2s0.300
> 
> IP 193.72.186.190.52636 > 193.72.186.130.8080 (normal)
> 
> IP 193.72.186.190.52636 > 192.168.202.10.80 (good, it was DNATted, BUT should be on enp2s0.202!)
> 
> 
> aka the DNAT is executed, but then 192.168.202.10 is not routed
> correctly. From the diagram https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks
> I thought that routing would be done AFTER prerouting/DNAT.
> 
> The routing table has:
> 
> default via 46.140.72.217 dev br0 onlink
> 46.140.72.216/29 dev br0 proto kernel scope link src 46.140.72.218
> 192.168.202.0/24 dev enp2s0.202 proto kernel scope link src 192.168.202.2
> 193.72.186.128/26 dev enp2s0.300 proto kernel scope link src 193.72.186.130
> 
> (*) not from 193.72.186.0/24, however, gets the same bug.
> 
> The nftables config:
> 
> table ip filter {
> # obviously a later goal is also to encode the L4 protocol here
> # and not hardcode it in the prerouting
> map multihoming_ext {
> type ipv4_addr . inet_service : ipv4_addr . inet_service
> elements = { 193.72.186.130 . 8080 : 192.168.202.10 . 80,
> 46.140.72.218 . 8080 : 192.168.202.10 . 80 }
> }
> 
> set w_all {
> type ipv4_addr
> flags interval
> elements = { 46.140.72.216/29,
> 192.168.202.1, 193.72.186.0/24
> }
> }
> 
> chain input {
> type filter hook input priority filter; policy drop;
> ct state invalid counter packets 3 bytes 120 drop
> ct state { established, related } counter packets 1092 bytes 76846 accept
> iif "lo" counter packets 0 bytes 0 accept
> counter packets 324 bytes 13539 jump whitelist
> counter packets 323 bytes 13455 jump blacknets
> counter packets 323 bytes 13455 jump blacklist
> counter packets 323 bytes 13455 jump incoming
> }
> 
> chain forward {
> type filter hook forward priority filter; policy drop;
> ct state invalid counter packets 0 bytes 0 drop
> ct state { established, related } counter packets 12 bytes 548 accept
> 
> # already after DNAT, obviously
> iifname "br0" ip daddr 192.168.202.10 tcp dport 80 accept
> iifname "enp2s0.300" ip daddr 192.168.202.10 tcp dport 80 accept
> }
> 
> chain output {
> type filter hook output priority filter; policy accept;
> ct state invalid counter packets 0 bytes 0 drop
> ct state { established, related } counter packets 890 bytes 182096 accept
> oif "lo" counter packets 0 bytes 0 accept
> counter packets 1 bytes 76 jump outgoing
> }
> 
> chain rejectcounter {
> meta l4proto tcp counter packets 0 bytes 0 reject with tcp reset
> meta l4proto udp counter packets 0 bytes 0 reject
> counter packets 0 bytes 0 drop
> }
> 
> chain dropcounter {
> counter packets 320 bytes 13349 drop
> }
> 
> chain whitelist {
> ip saddr @w_all counter packets 1 bytes 84 accept
> }
> 
> chain blacknets {
> }
> 
> chain blacklist {
> }
> 
> chain incoming {
> icmp type echo-request counter packets 3 bytes 106 accept
> icmp type echo-reply counter packets 0 bytes 0 accept
> tcp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
> udp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
> counter packets 320 bytes 13349 jump dropcounter
> }
> 
> chain outgoing {
> }
> 
> chain multihoming_prerouting {
> type nat hook prerouting priority dstnat; policy accept;
> 
> # this is the DNAT
> dnat ip to ip daddr . tcp dport map @multihoming_ext
> }
> }
> table ip myhelpers {
> chain prerouting {
> type filter hook prerouting priority filter; policy accept;
> }
> }
> 
> [ I removed ip6 entries ]
> 
> Any idea what could be wrong?
> 
> Thank you.
> 
> (**) very nice:
> br0:
> IP 46.140.72.222.60394 > 46.140.72.218.8080
> 
> IP 46.140.72.218.8080 > 46.140.72.222.60394
> 
> IP 46.140.72.222.60394 > 46.140.72.218.8080
> 
> 
> enp2s0.202 (another connection):
> IP 46.140.72.222.57790 > 192.168.202.10.80
> 
> IP 192.168.202.10.80 > 46.140.72.222.57790
> 
> IP 46.140.72.222.57790 > 192.168.202.10.80
> 
>

Re: nftables DNAT routes to wrong iface

[Marc SCHAEFER]

Hello,

On Sun, Jan 26, 2025 at 06:19:23AM +0000, Sunny73Cr wrote:
> > 193.72.186.128/26 dev enp2s0.300 proto kernel scope link src 193.72.186.130
> New reply (and now public): '193.72.186.128/26' does not cover all of
> '193.72.186.0/24', and some packets will not get routed.

Why should it?  The firewall 193.72.186.130 on which the DNAT is done is
connected to 193.72.186.128/26 because this is how the router
193.72.186.129 on the other side of the VLAN 300 defined that VLAN:

   193.72.186.128/26 dev enp5s0.300 proto kernel scope link src 193.72.186.129 

Indeed, the addresses 193.72.186.128 (reserved, subnet) to
193.72.186.191 (reserved, subnet broadcast) correspond to a
193.72.186.128/26 aka 32 - 26 = 6 free bit, so 64 addresses.

On the same VLANs there are other machines, such as 193.72.186.190 that
I used for my test.

BTW, those (193.72.186.129, 193.72.186.190) are ping-reachable from the
Internet (193.72.186.130 is currently down, as it is a test machine).
The setup works, it's just the DNAT that routes wrong for some
reason (I am an nftables beginner).  Do not try random connections to
those IP addresses (ping is ok), you might trigger the IDS.

The DNAT problem is that when a machine from 193.72.186.128/26 (e.g.
193.72.186.190 in the example) sends a TCP datagram on port 8080
of 193.72.186.130, this gets correctly DNATted to 192.168.202.10
port 80, however, then the routing entry:

> 192.168.202.0/24 dev enp2s0.202 proto kernel scope link src 192.168.202.2

is not respected, and the datagram is sent on enp2s0.300.

In another case, which is 46.140.72.218:8080 (on enp2s0.400), the DNAT
correctly works and sends to 192.168.202.10 on enp2s0.202.

As said in the mail, the reply packet won't work on 193.72.186.128/26,
that's expected (not default route, and no conntrack/mark set
yet), but it's the initial packet which is routed wrong.

Thank you & have a nice day.

Re: nftables DNAT routes to wrong iface

[Marc SCHAEFER]

Hello,

On Sat, Jan 25, 2025 at 04:55:49PM +0100, Marc SCHAEFER wrote:
>    telnet 193.72.186.130 8080 gives this on enp2s0.300
> 
>       IP 193.72.186.190.52636 > 193.72.186.130.8080 (normal)
>       IP 193.72.186.190.52636 > 192.168.202.10.80   (good, it was DNATted, BUT should be on enp2s0.202!)

Ok, my fault!

There still was an old configuration lying around which did:

root@test:~# ip rule
0:      from all lookup local
32764:  from 193.72.186.128/26 lookup 193
32766:  from all lookup main
32767:  from all lookup default

obviously, that's the bug.

If I remove those, then it works like it should.

I will now work on the conntrack part.