Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

[Swarm NameRedacted]

Hi,

I am trying to build a quick script via TC direct action and eBPF to
modify the destination IP of a packet so that it is routed through a
different bridge interface. Made a quick network diagram below to
demonstrate it. 

      Packet (dst: 10.10.3.2)
                |
                |
    ingress - (change dst to 10.10.4.1)
                |
                |
               eth0
                |
                |
      br0 - (addr: 10.10.3.1)
__eth0______   ___ens19_______
     |                |
     |                |
     |                |
     |                |
host: 10.10.4.1  host: 10.10.3.2



As shown, I send a packet from a separate client to eth0. eth0 is the
WAN interface of its machine and ens19 is the LAN interface; both are
connecting with bridge br0. Without modification, the packet goes
straight through ens19 to 10.10.3.2. 

Theoretically, by modifying the destination IP to 10.10.4.1 at ingress,
the packet should be rerouted to go back through eth0. However, in
practice, I find that the packet still goes through ens19 after
modification, and of course after that it never reaches anything. 

Why is it that ingress catches the packet before the bridging decision,
but the packet isn't rerouted? Is there a better way to do this?

Re: Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

[Andrew Lunn]

On Wed, Aug 05, 2020 at 08:19:57AM +0000, Swarm NameRedacted wrote:
> Hi,
> 
> I am trying to build a quick script via TC direct action and eBPF to
> modify the destination IP of a packet so that it is routed through a
> different bridge interface. Made a quick network diagram below to
> demonstrate it. 
> 
>       Packet (dst: 10.10.3.2)
>                 |
>                 |
>     ingress - (change dst to 10.10.4.1)
>                 |
>                 |
>                eth0
>                 |
>                 |
>       br0 - (addr: 10.10.3.1)
> __eth0______   ___ens19_______
>      |                |
>      |                |
>      |                |
>      |                |
> host: 10.10.4.1  host: 10.10.3.2
> 
> 
> 
> As shown, I send a packet from a separate client to eth0. eth0 is the
> WAN interface of its machine and ens19 is the LAN interface; both are
> connecting with bridge br0. Without modification, the packet goes
> straight through ens19 to 10.10.3.2. 
> 
> Theoretically, by modifying the destination IP to 10.10.4.1 at ingress,
> the packet should be rerouted to go back through eth0. However, in
> practice, I find that the packet still goes through ens19 after
> modification, and of course after that it never reaches anything. 
> 
> Why is it that ingress catches the packet before the bridging decision,
> but the packet isn't rerouted? Is there a better way to do this?
 
What is not clear is the subnet size. Is this all a /16 network? So
that 10.10.4.1 and 10.10.3.2 are in the same subnet? Or are these
different subnets, and you have a router somewhere you do not show in
your diagram? Is this redirect happening at L2, or L3? Maybe you even
have NAT involved, since you talk about WAN and LAN? Do you also need
to be modifying the destination MAC address?

You also need to think about at what layer in the stack is the IP
address being modified? A bridge works on L2. Is the packet being
bridges at L2 and sent out without an IP processing at L3? Do you
actually want to be using ebtables to modify the packet at L2?

But maybe take a step back. You are wanting to do something really
odd. What is your real use case. Maybe there is a better way to do
what you want. Please explain why you want to send a packet back out
the way it came in, with a different IP address.

	 Andrew

Re: Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

[Swarm NameRedacted]

All fair points, I'll address them one by one. 
1) The subnet size on everything is /16; everything is on the same
subnet (hence the bridge) except for the client which sends the initial
SYN packet. Modifying the destination MAC address was definitely
something I overlooked and that did get the packet running through the
correct interface. I got a bit thrown off that the bridge has it's own
MAC address that is identical to the LAN interface and couldn't
visualize it as an L2 switch. However, the packet is still being
dropped; I suspect it might be a checksum error but the only incorrect
checksum is TCP. Might have accidentally disabled checksum offloading. I'm not
sure

2) The destination IP address is being modified to 10.10.4.1 at L3. However, I
forgot to change the destination MAC to eth0 at L2 level as well, as you just
mentioned. 

3) The eventual goal is to build my own modified version of the SYNPROXY
protocol that includes some security enhancements in the packet itself.
Therefore, my current plan is to skip conntrack on the gateway
(10.10.3.1) for the packet initially destined to the server (10.10.3.2) 
and forward the packet to a separate server (10.10.4.1) (virtually configuring it 
so it is behind another transparent gateway (ip not relevant because the packet
should just pass right through) accessible via the current gateway). To make
things simple, I used a bridge on both gateways and am trying to route through
different interfaces on that bridge. 

Hope that makes sense. Please let me know if there's something I didn't clarify
fully that can be explained further!


On Wed, Aug 05, 2020 at 03:39:22PM +0200, Andrew Lunn wrote:
> 
> On Wed, Aug 05, 2020 at 08:19:57AM +0000, Swarm NameRedacted wrote:
> > Hi,
> >
> > I am trying to build a quick script via TC direct action and eBPF to
> > modify the destination IP of a packet so that it is routed through a
> > different bridge interface. Made a quick network diagram below to
> > demonstrate it.
> >
> >       Packet (dst: 10.10.3.2)
> >                 |
> >                 |
> >     ingress - (change dst to 10.10.4.1)
> >                 |
> >                 |
> >                eth0
> >                 |
> >                 |
> >       br0 - (addr: 10.10.3.1)
> > __eth0______   ___ens19_______
> >      |                |
> >      |                |
> >      |                |
> >      |                |
> > host: 10.10.4.1  host: 10.10.3.2
> >
> >
> >
> > As shown, I send a packet from a separate client to eth0. eth0 is the
> > WAN interface of its machine and ens19 is the LAN interface; both are
> > connecting with bridge br0. Without modification, the packet goes
> > straight through ens19 to 10.10.3.2.
> >
> > Theoretically, by modifying the destination IP to 10.10.4.1 at ingress,
> > the packet should be rerouted to go back through eth0. However, in
> > practice, I find that the packet still goes through ens19 after
> > modification, and of course after that it never reaches anything.
> >
> > Why is it that ingress catches the packet before the bridging decision,
> > but the packet isn't rerouted? Is there a better way to do this?
> 
> What is not clear is the subnet size. Is this all a /16 network? So
> that 10.10.4.1 and 10.10.3.2 are in the same subnet? Or are these
> different subnets, and you have a router somewhere you do not show in
> your diagram? Is this redirect happening at L2, or L3? Maybe you even
> have NAT involved, since you talk about WAN and LAN? Do you also need
> to be modifying the destination MAC address?
> 
> You also need to think about at what layer in the stack is the IP
> address being modified? A bridge works on L2. Is the packet being
> bridges at L2 and sent out without an IP processing at L3? Do you
> actually want to be using ebtables to modify the packet at L2?
> 
> But maybe take a step back. You are wanting to do something really
> odd. What is your real use case. Maybe there is a better way to do
> what you want. Please explain why you want to send a packet back out
> the way it came in, with a different IP address.
> 
> 	 Andrew

Re: Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

[Ido Schimmel]

On Wed, Aug 05, 2020 at 08:12:08PM +0000, Swarm NameRedacted wrote:
> All fair points, I'll address them one by one. 
> 1) The subnet size on everything is /16; everything is on the same
> subnet (hence the bridge) except for the client which sends the initial
> SYN packet. Modifying the destination MAC address was definitely
> something I overlooked and that did get the packet running through the
> correct interface. I got a bit thrown off that the bridge has it's own
> MAC address that is identical to the LAN interface and couldn't
> visualize it as an L2 switch. However, the packet is still being
> dropped; I suspect it might be a checksum error but the only incorrect
> checksum is TCP. Might have accidentally disabled checksum offloading. I'm not
> sure

You might need to enable hairpin on eth0:

# ip link set dev eth0 type bridge_slave hairpin on

Re: Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

[Swarm NameRedacted]

Not sure this applies. There's no NAT since everything is on the same
subnet. 

On Thu, Aug 06, 2020 at 09:33:36AM +0300, Ido Schimmel wrote:
> 
> On Wed, Aug 05, 2020 at 08:12:08PM +0000, Swarm NameRedacted wrote:
> > All fair points, I'll address them one by one.
> > 1) The subnet size on everything is /16; everything is on the same
> > subnet (hence the bridge) except for the client which sends the initial
> > SYN packet. Modifying the destination MAC address was definitely
> > something I overlooked and that did get the packet running through the
> > correct interface. I got a bit thrown off that the bridge has it's own
> > MAC address that is identical to the LAN interface and couldn't
> > visualize it as an L2 switch. However, the packet is still being
> > dropped; I suspect it might be a checksum error but the only incorrect
> > checksum is TCP. Might have accidentally disabled checksum offloading. I'm not
> > sure
> 
> You might need to enable hairpin on eth0:
> 
> # ip link set dev eth0 type bridge_slave hairpin on

Re: Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

[Ido Schimmel]

On Thu, Aug 06, 2020 at 07:00:15AM +0000, Swarm NameRedacted wrote:
> Not sure this applies. There's no NAT since everything is on the same
> subnet. 

IIUC, packet is received on eth0, you then change the DMAC to SMAC on
ingress (among other things) and then packet continues to the bridge.
The bridge checks the DMAC and sees that the packet is supposed to be
forwarded out of eth0. Since it's also the ingress interface the packet
is dropped. To overcome this you need to enable hairpin.

Re: Packet not rerouting via different bridge interface after modifying destination IP in TC ingress hook

[Swarm NameRedacted]

Ok that makes sense. I'll try it. However, doesn't hairpin imply that
the packet will be routed back into the same machine via some public
address and separate router? I'm just trying to redirect it to
10.10.4.1, not loop it back to 10.10.3.1. Or is this an
unorthodox/modified usage of hairpinning?

On Thu, Aug 06, 2020 at 10:49:09AM +0300, Ido Schimmel wrote:
> 
> On Thu, Aug 06, 2020 at 07:00:15AM +0000, Swarm NameRedacted wrote:
> > Not sure this applies. There's no NAT since everything is on the same
> > subnet.
> 
> IIUC, packet is received on eth0, you then change the DMAC to SMAC on
> ingress (among other things) and then packet continues to the bridge.
> The bridge checks the DMAC and sees that the packet is supposed to be
> forwarded out of eth0. Since it's also the ingress interface the packet
> is dropped. To overcome this you need to enable hairpin.