Redirect bridged traffic

Redirect bridged traffic

[Jaga Doe]

      Hello,

   I have a device with 3 NICs connected to two networks as follow:
1. eth0 connected to monitor network used for accessing my device and do configurations.
2. eth1 connected to main network which contains routers, DHCP server, NTP server, etc.
3. eth2 connected with a direct cable to a device (PC0) supposed to be part of the main network.

My device needs to pass through the entire traffic between PC0 and the main network excepting the one connecting on port 3000 of PC0 which needs to be redirected to a local process. The local process at it's turn needs to connect to PC0 on port 3000 and the replies to be passed back to the main network client.

For this I have created a bridge which is able to pass all the traffic between eth1 and eth2:
brctl addbr br0
brctl addif br0 eth1
brctl addif br0 eth2

ip link set br0 up

Using this, the PC0 is able to access all services from the main network (i.e DHCP for getting the IP address). Though the performances of the bridge is much lower than expected, the speed being almost 6x lower comparing to direct access of PC0 to main network.

For redirecting the traffic I was trying to use the nftables configuration below:

#!/usr/sbin/nft -f

flush ruleset

define MON_MAC=00:0c:29:15:7b:a0
define TPORT

table bridge tbrFilter {
    chain cbrRedirect {
      type filter hook prerouting priority 0;
      log tcp dport $TPORT meta pkttype set host ether daddr set $MON_MAC counter
    }
}

table inet tlcRedirect {
    chain clcRedirect {
      type nat hook prerouting priority 0;
      log tcp dport $TPORT counter redirect to $TPORT
    }
}

Using this configuration, the connection to PC0:3000 seems that it is captured by the nft but is not going to the local process.

What I am missing here?

Thanks and best regards,
Jaga.

Re: Redirect bridged traffic

[Florian Westphal]

Jaga Doe <jaga.doe@aol.com> wrote:
> table bridge tbrFilter {
>     chain cbrRedirect {
>       type filter hook prerouting priority 0;
>       log tcp dport $TPORT meta pkttype set host ether daddr set $MON_MAC counter
>     }
> }
> 
> table inet tlcRedirect {
>     chain clcRedirect {
>       type nat hook prerouting priority 0;
>       log tcp dport $TPORT counter redirect to $TPORT
>     }
> }
> 
> Using this configuration, the connection to PC0:3000 seems that it is captured by the nft but is not going to the local process.
> 
> What I am missing here?

I don't see anything wrong here, this should work and $TPORT should end
up in inet input hook.
Does it end up in forward instead or does it 'just disappear'?

Does 'nstat' show anything?

Re: Redirect bridged traffic

[Jaga Doe]

      Hello,

   I have discovered a great mechanism of understanding what is happening, but setting nftrace flag. With that I can see the dataflow:

trace id 9af9e0b6 bridge tbrFilter cbrRedirect packet: iif "eth1" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
trace id 9af9e0b6 bridge tbrFilter cbrRedirect rule log tcp dport 3000 meta pkttype set host ether daddr set 00:0c:29:15:7b:a0 counter packets 5 bytes 300 meta nftrace set 1 (verdict continue)
trace id 9af9e0b6 bridge tbrFilter cbrRedirect verdict continue 
trace id 9af9e0b6 bridge tbrFilter cbrRedirect policy accept 
trace id 9af9e0b6 inet tlcRedirect clcRedirect packet: iif "br0" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
trace id 9af9e0b6 inet tlcRedirect clcRedirect rule log tcp dport 3000 counter packets 5 bytes 300 redirect to :3000 (verdict drop)

I don't understand why the redirect rule is leading to the drop verdict.

Thank you,
Jaga.


-----Original Message-----
From: Florian Westphal <fw@strlen.de>
To: Jaga Doe <jaga.doe@aol.com>
Cc: netfilter <netfilter@vger.kernel.org>
Sent: Wed, Feb 5, 2020 9:35 pm
Subject: Re: Redirect bridged traffic

Jaga Doe <jaga.doe@aol.com> wrote:
> table bridge tbrFilter {
>     chain cbrRedirect {
>       type filter hook prerouting priority 0;
>       log tcp dport $TPORT meta pkttype set host ether daddr set $MON_MAC counter
>     }
> }
> 
> table inet tlcRedirect {
>     chain clcRedirect {
>       type nat hook prerouting priority 0;
>       log tcp dport $TPORT counter redirect to $TPORT
>     }
> }
> 
> Using this configuration, the connection to PC0:3000 seems that it is captured by the nft but is not going to the local process.
> 
> What I am missing here?

I don't see anything wrong here, this should work and $TPORT should end
up in inet input hook.
Does it end up in forward instead or does it 'just disappear'?

Does 'nstat' show anything?

Re: Redirect bridged traffic

[Florian Westphal]

Jaga Doe <jaga.doe@aol.com> wrote:
>       Hello,
> 
>    I have discovered a great mechanism of understanding what is happening, but setting nftrace flag. With that I can see the dataflow:
> 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect packet: iif "eth1" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect rule log tcp dport 3000 meta pkttype set host ether daddr set 00:0c:29:15:7b:a0 counter packets 5 bytes 300 meta nftrace set 1 (verdict continue)
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect verdict continue 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect policy accept 
> trace id 9af9e0b6 inet tlcRedirect clcRedirect packet: iif "br0" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
> trace id 9af9e0b6 inet tlcRedirect clcRedirect rule log tcp dport 3000 counter packets 5 bytes 300 redirect to :3000 (verdict drop)
> 
> I don't understand why the redirect rule is leading to the drop verdict.

Most simple explanation: your bridge doesn't have an ip address
(redirect is really just 'dnat to' with the primary address of the
 incoming interface).

Re: Redirect bridged traffic

[Jaga Doe]

Hey! Of course that the issue was in my side: a bad rule was messing up the nftables. Trying to make my device 'invisible' I was adding a rule meant to change the ip/mac of the package, but I was using dnat instead of snat.

Sorry if I have waste your time


-----Original Message-----
From: Florian Westphal <fw@strlen.de>
To: Jaga Doe <jaga.doe@aol.com>
Cc: fw <fw@strlen.de>; netfilter <netfilter@vger.kernel.org>
Sent: Thu, Feb 6, 2020 10:46 am
Subject: Re: Redirect bridged traffic

Jaga Doe <jaga.doe@aol.com> wrote:
>       Hello,
> 
>    I have discovered a great mechanism of understanding what is happening, but setting nftrace flag. With that I can see the dataflow:
> 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect packet: iif "eth1" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect rule log tcp dport 3000 meta pkttype set host ether daddr set 00:0c:29:15:7b:a0 counter packets 5 bytes 300 meta nftrace set 1 (verdict continue)
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect verdict continue 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect policy accept 
> trace id 9af9e0b6 inet tlcRedirect clcRedirect packet: iif "br0" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
> trace id 9af9e0b6 inet tlcRedirect clcRedirect rule log tcp dport 3000 counter packets 5 bytes 300 redirect to :3000 (verdict drop)
> 
> I don't understand why the redirect rule is leading to the drop verdict.

Most simple explanation: your bridge doesn't have an ip address
(redirect is really just 'dnat to' with the primary address of the
 incoming interface).

Re: Redirect bridged traffic

[Jaga Doe]

You are totally right. Adding an IP address to the bridge, is passing the redirect drop stage. I thought that having all SRC/DST parameters set in the package, is supposed to have the data flow completed.
However. after adding the bridge IP, the package seems that is getting lost in another step. In order to troubleshoot the issue I have created all hooks for all chains in ip, inet and bridge tables. For understanding easier where the package is I have created the tables with the naming tb_<table type> and rules ch_<chain type>_<hook>.

 The nftrace feature is allowing me to follow how the package is traversing the network stack:

trace id ff3b1d8d bridge tb_bridge ch_filter_prerouting packet: iif "eth1" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 64068 ip protocol tcp ip length 60 tcp sport 46206 tcp dport 3000 tcp flags == syn tcp window 64240 
trace id ff3b1d8d bridge tb_bridge ch_filter_prerouting rule log tcp dport 3000 meta pkttype set host ether daddr set 00:0c:29:15:7b:a0 counter packets 8 bytes 480 meta nftrace set 1 (verdict continue)
trace id ff3b1d8d bridge tb_bridge ch_filter_prerouting rule log tcp dport 3000 comment "bridge-filter-prerouting" (verdict continue)
trace id ff3b1d8d bridge tb_bridge ch_filter_prerouting verdict continue 
trace id ff3b1d8d bridge tb_bridge ch_filter_prerouting policy accept 
trace id ff3b1d8d inet tb_inet ch_nat_prerouting packet: iif "br0" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 64068 ip protocol tcp ip length 60 tcp sport 46206 tcp dport 3000 tcp flags == syn tcp window 64240 
trace id ff3b1d8d inet tb_inet ch_nat_prerouting rule log tcp dport 3000 counter packets 8 bytes 480 redirect to :3000 (verdict accept)
trace id ff3b1d8d inet tb_inet ch_filter_prerouting packet: iif "br0" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.100.10 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 64068 ip protocol tcp ip length 60 tcp sport 46206 tcp dport 3000 tcp flags == syn tcp window 64240 
trace id ff3b1d8d inet tb_inet ch_filter_prerouting rule log tcp dport 3000 comment "inet-filter-prerouting" (verdict continue)
trace id ff3b1d8d inet tb_inet ch_filter_prerouting verdict continue 
trace id ff3b1d8d inet tb_inet ch_filter_prerouting policy accept 
trace id ff3b1d8d ip tb_ip ch_filter_prerouting packet: iif "br0" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.100.10 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 64068 ip length 60 tcp sport 46206 tcp dport 3000 tcp flags == syn tcp window 64240 
trace id ff3b1d8d ip tb_ip ch_filter_prerouting rule log tcp dport 3000 comment "ip-filter-prerouting" (verdict continue)
trace id ff3b1d8d ip tb_ip ch_filter_prerouting verdict continue 
trace id ff3b1d8d ip tb_ip ch_filter_prerouting policy accept 

From this capture I can see that the package is going throuh prerouting in all tables, the last one being ip table. Here even though the verdict is continue and the policy accept, the package cannot be seen in other hooks input, forward, output or postrouting (therefore the application listening locally to 3000 is not getting the connection request). It seems that the package is simply vanishing.

The nstat is returning:
#3661.1804289383 sampling_interval=10 time_const=60
IpInReceives                    66                 6.6
IpInAddrErrors                  0                  0.4
IpInDelivers                    53                 4.6
IpOutRequests                   52                 4.5
IpOutNoRoutes                   1                  0.1
TcpInSegs                       53                 4.6
TcpOutSegs                      56                 4.7
TcpRetransSegs                  2                  0.1
Ip6InReceives                   1                  0.1
Ip6InMcastPkts                  1                  0.1
Ip6InOctets                     72                 6.4
Ip6OutOctets                    0                  0.6
Ip6InMcastOctets                72                 6.3
Ip6OutMcastOctets               0                  0.6
Ip6InNoECTPkts                  1                  0.1
TcpExtDelayedACKs               1                  0.1
TcpExtTCPHPHits                 0                  0.3
TcpExtTCPPureAcks               14                 0.9
TcpExtTCPHPAcks                 32                 2.7
TcpExtTCPLossProbes             3                  0.1
TcpExtTCPBacklogCoalesce        1                  0.1
TcpExtTCPDSACKRecv              2                  0.1
TcpExtIPReversePathFilter       10                 1.2
TcpExtTCPAutoCorking            12                 1.2
TcpExtTCPOrigDataSent           52                 4.3
TcpExtTCPDelivered              54                 4.4
IpExtInOctets                   4275             526.9
IpExtOutOctets                  23132           1469.2
IpExtInBcastOctets              0                  4.4
IpExtInNoECTPkts                66                 6.6

Any idea on how to troubleshoot?

Thanks,
Jaga

-----Original Message-----
From: Florian Westphal <fw@strlen.de>
To: Jaga Doe <jaga.doe@aol.com>
Cc: fw <fw@strlen.de>; netfilter <netfilter@vger.kernel.org>
Sent: Thu, Feb 6, 2020 10:46 am
Subject: Re: Redirect bridged traffic

Jaga Doe <jaga.doe@aol.com> wrote:
>       Hello,
> 
>    I have discovered a great mechanism of understanding what is happening, but setting nftrace flag. With that I can see the dataflow:
> 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect packet: iif "eth1" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect rule log tcp dport 3000 meta pkttype set host ether daddr set 00:0c:29:15:7b:a0 counter packets 5 bytes 300 meta nftrace set 1 (verdict continue)
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect verdict continue 
> trace id 9af9e0b6 bridge tbrFilter cbrRedirect policy accept 
> trace id 9af9e0b6 inet tlcRedirect clcRedirect packet: iif "br0" ether saddr f4:8c:50:fa:1f:e5 ether daddr 00:0c:29:15:7b:a0 ip saddr 192.168.0.21 ip daddr 192.168.0.123 ip dscp 0x04 ip ecn not-ect ip ttl 64 ip id 18388 ip protocol tcp ip length 60 tcp sport 40902 tcp dport 3000 tcp flags == syn tcp window 64240 
> trace id 9af9e0b6 inet tlcRedirect clcRedirect rule log tcp dport 3000 counter packets 5 bytes 300 redirect to :3000 (verdict drop)
> 
> I don't understand why the redirect rule is leading to the drop verdict.

Most simple explanation: your bridge doesn't have an ip address
(redirect is really just 'dnat to' with the primary address of the
 incoming interface).