snat bridge routes reply packets

snat bridge routes reply packets

[Amin Azez]

I have a bridge setup snat'ing some IP's that aren't actually on the
bridge subnet.

That works fine except that the nat code doesn't know what to do with
the de-snat'd packets and tries to send them to the default gateway.

This makes sense, but it's routing, not bridging.

Bridging dnat would keep track of the physdev and src mac as well as the
src ip and ignore arp altogether when re-issueing the de-snat'd
responses; it would write the de-snat'd packet back out to the original
interface and targeted to the original mac.

I think bridging snat can be detected when the target mac of the packet
that causes the conntrack to be created does not match the mac of the
physdev the packet comes in on (i.e. the snat'ing box is not being used
as a gateway).

In such cases the physdev and srcmac must be saved.

(Co-incidentally I'm already doing this in my contrack mods which will
no doubt get merged into nf_conntrack that does some layer 2 stuff)

So my real questions are:
1) Am I nuts?
2) Any tips on skipping the arp cache stuff on the de-snat'd response,
as I have a reference to the physdev and target mac already available?

I also think there is some mileage in checking the src_mac when looking
up conntracks in the orig direction, to avoid having to presume that the
same ip address does not exist on different physdev of the same bridge -
to thus allow them both to be snat'd independantly.

Sam Azez

Re: snat bridge routes reply packets

[Henrik Nordstrom]

On Wed, 28 Sep 2005, Amin Azez wrote:

> I have a bridge setup snat'ing some IP's that aren't actually on the
> bridge subnet.
>
> That works fine except that the nat code doesn't know what to do with
> the de-snat'd packets and tries to send them to the default gateway.
>
> This makes sense, but it's routing, not bridging.

As soon as you NAT a packet in bridge-netfilter it is intercepted and 
given to routing, no longer bridged. This to allow REDIRECT and a number 
of other highly interesting applications of NAT.

Yes, this causes a bit of headache if your bridge does not have correct 
routing, but it's a fairly small price to pay for a wealth of interesting 
functionality which would not be possible if the bridge did not reroute 
packets after applying NAT.

At a first glance one could think that SNAT would not need to do this, but 
then you realize that by SNAT:in you break the IP-MAC relation of the 
traffic flow, especially in the return direction which is a DNAT 
operation.  Doing DNAT without rerouting does not make much sense as the 
packets would then still be delivered to the same destination MAC, and 
things like REDIRECT would not be possible.

> 1) Am I nuts?

No. Just on a different angle than how the bridge-netfilter integration is 
designed wrt NAT.

Regards
Henrik

Re: snat bridge routes reply packets

[Amin Azez]

Thanks for your interesting response;

Henrik Nordstrom wrote:
> On Wed, 28 Sep 2005, Amin Azez wrote:
> 
>> I have a bridge setup snat'ing some IP's that aren't actually on the
>> bridge subnet.
>>
>> That works fine except that the nat code doesn't know what to do with
>> the de-snat'd packets and tries to send them to the default gateway.
>>
>> This makes sense, but it's routing, not bridging.
> 
> As soon as you NAT a packet in bridge-netfilter it is intercepted and
> given to routing, no longer bridged. This to allow REDIRECT and a number
> of other highly interesting applications of NAT.
> 
> Yes, this causes a bit of headache if your bridge does not have correct
> routing, but it's a fairly small price to pay for a wealth of
> interesting functionality which would not be possible if the bridge did
> not reroute packets after applying NAT.

True enough, the outward bound snat'd packet is routed. The bridge does
have correct routing for the snat'd items.

> At a first glance one could think that SNAT would not need to do this,
> but then you realize that by SNAT:in you break the IP-MAC relation of
> the traffic flow, especially in the return direction which is a DNAT
> operation.  Doing DNAT without rerouting does not make much sense as the
> packets would then still be delivered to the same destination MAC, and
> things like REDIRECT would not be possible.

I am curious about this.

It is a change of the response dnat that I am suggesting. The DNAT
packets need delivering to the original MAC/IP/PHYSDEV in bridging mode,
and in routing mode the routing stack will probably find the same
MAC/IP/PHYSDEV using its routing tricks and ARP. In bridging mode the
routing code cannot find this.

This is, I admin, using iptables as a partial bridge/router when
snat'ing; however the outward packets ARE handled fine, I want to fix up
the return packets.

>> 1) Am I nuts?
> 
> 
> No. Just on a different angle than how the bridge-netfilter integration
> is designed wrt NAT.

Heh.  I think I could permit the reverse snat'ing in bridge mode without
damaging the other behaviour.

I was planning to use whether the target mac of the first packet matched
the mac of the incoming physdev as an indicator of bridging or routing,
but then I wondered if the interface merely being in promiscuous mode
would cause trouble, but I don't think so; but maybe I should do IP
checks instead and bridging is where the dest IP address of the first
packet doesn't match the ip address of the devices (not physdev) it came
in on?

Anyway, although this is not how the bridge-netfilter integration was
planned, do you think my proposal is sound?

Sam

Re: snat bridge routes reply packets

[Henrik Nordstrom]

On Thu, 29 Sep 2005, Amin Azez wrote:

> It is a change of the response dnat that I am suggesting. The DNAT
> packets need delivering to the original MAC/IP/PHYSDEV in bridging mode,
> and in routing mode the routing stack will probably find the same
> MAC/IP/PHYSDEV using its routing tricks and ARP. In bridging mode the
> routing code cannot find this.

More likely ARP is your problem.

When you SNAT to another address on the LAN you need to make sure there is 
something answering to ARP on that address, if not the receiving station 
won't be able to ARP for the MAC where to send return traffic.

> Heh.  I think I could permit the reverse snat'ing in bridge mode without
> damaging the other behaviour.

If the problem is what I suspect then "fixing" the NAT to not route the 
traffic won't make much difference.

> I was planning to use whether the target mac of the first packet matched
> the mac of the incoming physdev as an indicator of bridging or routing,

this is not sufficient and would break REDIRECT.

> but then I wondered if the interface merely being in promiscuous mode
> would cause trouble, but I don't think so; but maybe I should do IP
> checks instead and bridging is where the dest IP address of the first
> packet doesn't match the ip address of the devices (not physdev) it came
> in on?

this too would break REDIRECT.

the thing with NAT:ing in the bridge is that you change the IP addressing 
of the packet. As the IP addressing is changed there is a very high 
probability it needs to be rerouted. Bridgeing and preserving the MAC is 
in many cases not correct.

Yes, in MASQUERADE type flows where the firewall SNATs flows to it's own 
address it might be possible in certain cases to 
remember the original "client" MAC address, and avoid the ARP:ing on 
return traffic. But this is on the assumption that the client network is 
symmetric. It is possible the traffic was received via an assymetric route 
where return traffic should go to another MAC address and ARP is required 
to figure this out.

> Anyway, although this is not how the bridge-netfilter integration was
> planned, do you think my proposal is sound?

I see only two approaches. Either the current behaviour relying on routing 
being correctly configured, or an explicit flag in each NAT rule telling 
if the packets in this session should be rerouted or not moving the burden 
to the firewall administrator. Actually this flag might be good for 
iptables in general to optimize NAT performance slightly and not only when 
using bridge-netfilter. The constant rerouting of packets in a NAT session 
is a significant overhead and often it can be priorly known (by the admin) 
that this rerouting is not needed for the traffic flows in question.

But I strongly suspect your problems is not at all related to routing. It 
would only be routing related if your bridge does not have correct routing 
info for either the source or destination. If you have routing for both 
source and destination then your problem is almost certainly not routing 
related but most likely ARP related.

Regards
Henrik

Re: snat bridge routes reply packets

[Amin Azez]

Henrik Nordstrom wrote:
> On Thu, 29 Sep 2005, Amin Azez wrote:
> 
>> It is a change of the response dnat that I am suggesting. The DNAT
>> packets need delivering to the original MAC/IP/PHYSDEV in bridging mode,
>> and in routing mode the routing stack will probably find the same
>> MAC/IP/PHYSDEV using its routing tricks and ARP. In bridging mode the
>> routing code cannot find this.
> 
> ./net/ipv4/ipvs/ip_vs_proto_tcp.c:tcp_snat_handler
> More likely ARP is your problem.
> 
> When you SNAT to another address on the LAN you need to make sure there
> is something answering to ARP on that address, if not the receiving
> station won't be able to ARP for the MAC where to send return traffic.

Precicely, ARP is the problem. Because the de-nat'd return packet of the
snat'd flow is not on any local bridge subnets, no arp requests are
made. There are also other reasons for preferring not to make arp requests.

>> Heh.  I think I could permit the reverse snat'ing in bridge mode without
>> damaging the other behaviour.
> 
> If the problem is what I suspect then "fixing" the NAT to not route the
> traffic won't make much difference.

I'm still having difficulty seeing this. I only want to stop routing out
of the reply when it was bridged in rather than routed in.

>> I was planning to use whether the target mac of the first packet matched
>> the mac of the incoming physdev as an indicator of bridging or routing,
> 
> this is not sufficient and would break REDIRECT.

How would it do that?

>> but then I wondered if the interface merely being in promiscuous mode
>> would cause trouble, but I don't think so; but maybe I should do IP
>> checks instead and bridging is where the dest IP address of the first
>> packet doesn't match the ip address of the devices (not physdev) it came
>> in on?./net/ipv4/ipvs/ip_vs_proto_tcp.c:tcp_snat_handler
> 
> this too would break REDIRECT.
> 
> the thing with NAT:ing in the bridge is that you change the IP
> addressing of the packet. As the IP addressing is changed there is a
> very high probability it needs to be rerouted. Bridgeing and preserving
> the MAC is in many cases not correct.

I still want to route on the output of the snat'd packet, I just want to
avoid routing on the de-nat'd return packets.  I'm not sure how this
breaks redirects.

> Yes, in MASQUERADE type flows where the firewall SNATs flows to it's own
> address it might be possible in certain cases to remember the original
> "client" MAC address, and avoid the ARP:ing on return traffic. But this
> is on the assumption that the client network is symmetric. It is
> possible the traffic was received via an assymetric route where return
> traffic should go to another MAC address and ARP is required to figure
> this out.

Yes, this is a strong assumption, we would only do this when snat-ing
bridged traffic. There may be assymetric bridging which would break, but
then snat is already broken for bridging.

>> Anyway, although this is not how the bridge-netfilter integration was
>> planned, do you think my proposal is sound?
> 
> I see only two approaches. Either the current behaviour relying on
> routing being correctly configured, or an explicit flag in each NAT rule
> telling if the packets in this session should be rerouted or not moving
> the burden to the firewall administrator. Actually this flag might be
> good for iptables in general to optimize NAT performance slightly and
> not only when using bridge-netfilter. The constant rerouting of packets
> in a NAT session is a significant overhead and often it can be priorly
> known (by the admin) that this rerouting is not needed for the traffic
> flows in question.
> 

This sounds like a good solution, and in my preferred deployment an
option of the flag would be to "guess" whether or not to route the
return packet based on my stated criteria.

> But I strongly suspect your problems is not at all related to routing.
> It would only be routing related if your bridge does not have correct
> routing info for either the source or destination. 

My bridge does may not have routing for the source in many instances.

> If you have routing
> for both source and destination then your problem is almost certainly
> not routing related but most likely ARP related.

quite so.

Thanks for your helpful input. The matter of assymetric routes had not
applied to me, but I don't think it will be a problem, I only wish to
change behaviour where input routing was not happening.

Sam

Re: snat bridge routes reply packets

[Martijn Lievaart]

Amin Azez zei:
> Henrik Nordstrom wrote:
>> But I strongly suspect your problems is not at all related to routing.
>> It would only be routing related if your bridge does not have correct
>> routing info for either the source or destination.
>
> My bridge does may not have routing for the source in many instances.

Maybe a stupid remark, but if you create routes for the source to the
existing IP where you want the packet delivered? Does that solve your
problem?

M4

Re: snat bridge routes reply packets

[Amin Azez]

Martijn Lievaart wrote:
> Amin Azez zei:
> 
>>Henrik Nordstrom wrote:
>>
>>>But I strongly suspect your problems is not at all related to routing.
>>>It would only be routing related if your bridge does not have correct
>>>routing info for either the source or destination.
>>
>>My bridge does may not have routing for the source in many instances.
> 
> 
> Maybe a stupid remark, but if you create routes for the source to the
> existing IP where you want the packet delivered? Does that solve your
> problem?

My actual problem is that I need a bridging kernel that can be deployed
in unknown network environments and nat to a known gateway that is the
only machine guaranteed to be on the same subnet. It's not pretty.

We have created broad network aliases for the bridge so that all IP
addresses are local and roughly get the scenario you speak of, but then
we need to add static arp entries to assign the default gateway's mac to
specific known non-local ip, which is of course a worse hack than
mending snat for source-bridge scenarios.

Sam

Re: snat bridge routes reply packets

[Henrik Nordstrom]

On Thu, 29 Sep 2005, Amin Azez wrote:

> Precicely, ARP is the problem. Because the de-nat'd return packet of the
> snat'd flow is not on any local bridge subnets, no arp requests are
> made. There are also other reasons for preferring not to make arp requests.

So you do not have routing for the initiating host? Or what is your 
problem more exacly?

> I'm still having difficulty seeing this. I only want to stop routing out
> of the reply when it was bridged in rather than routed in.

There is not really any difference in request/reply to the bridge. Both 
are just packets. One is SNAT:ed, the other DNAT:ed. But it is true that 
conntrack knows reasonably well which direction the packet is flowing 
assuming "normal" traffic flows so it is not impossible.

>>> I was planning to use whether the target mac of the first packet matched
>>> the mac of the incoming physdev as an indicator of bridging or routing,
>>
>> this is not sufficient and would break REDIRECT.
>
> How would it do that?

If you do not reroute DNAT:ed packets then the bridge would deliver the 
packet to the original destination MAC, not to the local host.

Same applies if you DNAT within the local subnet.

> I still want to route on the output of the snat'd packet, I just want to
> avoid routing on the de-nat'd return packets.  I'm not sure how this
> breaks redirects.

For the same reasons the return packets needs to be routed. These packets 
are implicitly DNAT:ed, changing the destination IP to a different IP than 
the sending host ARP:ed for.

>> Yes, in MASQUERADE type flows where the firewall SNATs flows to it's own
>> address it might be possible in certain cases to remember the original
>> "client" MAC address, and avoid the ARP:ing on return traffic. But this
>> is on the assumption that the client network is symmetric. It is
>> possible the traffic was received via an assymetric route where return
>> traffic should go to another MAC address and ARP is required to figure
>> this out.
>
> Yes, this is a strong assumption, we would only do this when snat-ing
> bridged traffic. There may be assymetric bridging which would break, but
> then snat is already broken for bridging.

I would not agree. Both SNAT and DNAT works just fine in bridgeing 
assuming the IP level is properly set up to support the addresses used.

You can not SNAT to an otherwise non-existing IP as there then is no MAC 
address for the receiving host to sent the replies to (no ARP answer when 
the receiving host wants to send reply packets, so it can't).

> This sounds like a good solution, and in my preferred deployment an
> option of the flag would be to "guess" whether or not to route the
> return packet based on my stated criteria.

This "guess" must be done by the administrator when creating the rule, if 
not it gets very confusing.

Thinking this over there actually needs to be several different modes

   - Don't reroute the packet at all. All MAC addressing is preserved as-is 
on each packet. (relevant to both normal IP level iptables and bridgeing)

   - Track the initiating source MAC address, and reroute at MAC level all 
packets in the reply direction to this address. (only relevant to 
bridgeing). But route packets in the forward direction. **

   - Reroute all NAT:ed packets. (the default, what we have today).

One drawback of the proposed solution is that conntrack needs to be 
extended with MAC addressing in each direction.

**) This is only meaningful in MASQUERADE type SNAT setups where the 
bridge does not have routing to all "internal clients".


Regards
Henrik

Re: snat bridge routes reply packets

[Martijn Lievaart]

Amin Azez zei:

> We have created broad network aliases for the bridge so that all IP
> addresses are local and roughly get the scenario you speak of, but then
> we need to add static arp entries to assign the default gateway's mac to
> specific known non-local ip, which is of course a worse hack than
> mending snat for source-bridge scenarios.

Why would creating routes for those specific non-local IPs to that gateway
not work?

I still  fail to see what you are trying to achieve. In short, you SNAT
packets. You don't want to send the return packet to the original
source-IP based on normal ip-routing. Instead, you want to send it to the
MAC adres it was originally addressed to. I fail to see the value of this,
so I think I'm still missing something.

However, you can still achieve this by adding a route for the original
source IP to the IP it gets natted to. This would bypass the arp mechanism
and instead route the packet to the natted IP.

HTH,
M4

Re: snat bridge routes reply packets

[Henrik Nordstrom]

On Thu, 29 Sep 2005, Martijn Lievaart wrote:

> I still  fail to see what you are trying to achieve. In short, you SNAT
> packets. You don't want to send the return packet to the original
> source-IP based on normal ip-routing. Instead, you want to send it to the
> MAC adres it was originally addressed to. I fail to see the value of this,
> so I think I'm still missing something.


Simplest setup explaining the case as I understood it:

unknown network mesh -> Bridge running SNAT -> Internet

and you want the bridge to SNAT whatever is seen from the internal 
network, no matter what address is being used. The bridge does not have 
full knowledge of how the internal network mesh looks like, only that 
packets coming in on that interface is from the internal network.

Today this works fine for packets coming from the internal network going 
towards the Internet, but fails on return traffic as the bridge does not 
know where to route the "client" traffic.


If connection tracking in this kind of setup remembered the original 
source MAC of the incoming packet then this could be used in the implicit 
DNAT of the return traffic to route the packets (at level 2, not IP) to 
the sending host/router on the internal network mesh without requiring 
full routing information on the bridge.

This is in some senses similar to the level 3 routing without routing 
tables problem triggering my development of CONNMARK years ago. CONNMARK 
was initially triggered by the need to have a Internet gateway/firewall 
without any detailed knowledge of what the internal network looked like 
except for a router address where internal traffic could be sent.


Quite likely the problem Amin Azez is trying to address can be solved more 
cleanly by proxy-ARP + CONNMARK based policy routing than by bridge 
firewalling, but without knowing/understanding the exact layout of his 
network mess it's hard to say what it best. But I know from experience 
that bridge firewalling with NAT is very rarely the best approach in the 
long run, only perhaps the quickest if you are in a pinch with very 
limited time or prevented of doing it right.

Regards
Henrik

Re: snat bridge routes reply packets

[Martijn Lievaart]

Henrik Nordstrom wrote:

> On Thu, 29 Sep 2005, Martijn Lievaart wrote:
>
>> so I think I'm still missing something.
>
>
> Simplest setup explaining the case as I understood it:
>
> unknown network mesh -> Bridge running SNAT -> Internet
>
> and you want the bridge to SNAT whatever is seen from the internal 
> network, no matter what address is being used. The bridge does not 
> have full knowledge of how the internal network mesh looks like, only 
> that packets coming in on that interface is from the internal network.
>
>

Ah, now I see. The unknown network mesh uses multiple routers which are 
unknown to the bridge, so the bridge does not know where to route the 
return packets, although it could know if it saved the mac the original 
packet came from. Right?

M4

Re: snat bridge routes reply packets

[Amin Azez]

Henrik Nordstrom wrote:
> Quite likely the problem Amin Azez is trying to address can be solved
> more cleanly by proxy-ARP + CONNMARK based policy routing than by bridge
> firewalling, but without knowing/understanding the exact layout of his
> network mess it's hard to say what it best. 

Good points; the only reason the network-mess gets to bridge to the
router is that the bridge is arp-faking with ebtables to give out the
router mac address as if it were whatever the network-mess was arping
for. No doubt connmark routing would work too, I hadn't considered this;
but marks are already well used for other things.

> But I know from experience
> that bridge firewalling with NAT is very rarely the best approach in the
> long run, only perhaps the quickest if you are in a pinch with very
> limited time or prevented of doing it right.

Where the network mess cna be known in advance, I would agree, but in a
case where it "just needs to work" extending snat to bridging makes sense.

Anyway, I am proceeding with this, and appreciate the feedback. I shall
extend the snat iptables target to allow this as an optional extension.

Amin

Re: snat bridge routes reply packets

[Amin Azez]

Henrik Nordstrom wrote:
> If you do not reroute DNAT:ed packets then the bridge would deliver the
> packet to the original destination MAC, not to the local host.
> 
> Same applies if you DNAT within the local subnet.

I probably didn't make clear that I was looking at changing kernel
behaviour, not looking for iptables rules.

Oh; well that was the bit I was going to change. One of the bridgeing
code output functions (whose name I forget for now) was the trick;

>>> Yes, in MASQUERADE type flows where the firewall SNATs flows to it's own
>>> address it might be possible in certain cases to remember the original
>>> "client" MAC address, and avoid the ARP:ing on return traffic. But this
>>> is on the assumption that the client network is symmetric. It is
>>> possible the traffic was received via an assymetric route where return
>>> traffic should go to another MAC address and ARP is required to figure
>>> this out.
>>
>>
>> Yes, this is a strong assumption, we would only do this when snat-ing
>> bridged traffic. There may be assymetric bridging which would break, but
>> then snat is already broken for bridging.
> 
> 
> I would not agree. Both SNAT and DNAT works just fine in bridgeing
> assuming the IP level is properly set up to support the addresses used.

I agree, but pure bridging also works without this assumption and it
could work with snat bridging.

> You can not SNAT to an otherwise non-existing IP as there then is no MAC
> address for the receiving host to sent the replies to (no ARP answer
> when the receiving host wants to send reply packets, so it can't).

This is the problem I intend to solve

>   - Don't reroute the packet at all. All MAC addressing is preserved
> as-is on each packet. (relevant to both normal IP level iptables and
> bridgeing)
> 
>   - Track the initiating source MAC address, and reroute at MAC level
> all packets in the reply direction to this address. (only relevant to
> bridgeing). But route packets in the forward direction. **

I did say in my first post that my conntrack mods are already tracking
the mac address and phsedv of the originating packet; this is what I was
planning to do.

> 
>   - Reroute all NAT:ed packets. (the default, what we have today).
> 
> One drawback of the proposed solution is that conntrack needs to be
> extended with MAC addressing in each direction.

already got it, and afact nf_netfilter already does much in this
direction anway.

Sam

Re: snat bridge routes reply packets

[Henrik Nordstrom]

On Fri, 30 Sep 2005, Martijn Lievaart wrote:

> Ah, now I see. The unknown network mesh uses multiple routers which are 
> unknown to the bridge, so the bridge does not know where to route the return 
> packets, although it could know if it saved the mac the original packet came 
> from. Right?

This is my understanding of the original question, yes.

Regards
Henrik

Re: snat bridge routes reply packets

[Amin Azez]

Martijn Lievaart wrote:

>
> Ah, now I see. The unknown network mesh uses multiple routers which
> are unknown to the bridge, so the bridge does not know where to route
> the return packets, although it could know if it saved the mac the
> original packet came from. Right?

Yes. And having saved the mac address (and physdev) it will use the
bridge output functions instead of the ip route functions.

Sam

Re: snat bridge routes reply packets

[Henrik Nordstrom]

On Mon, 3 Oct 2005, Amin Azez wrote:

> Yes. And having saved the mac address (and physdev) it will use the
> bridge output functions instead of the ip route functions.

Saving the MAC should be sufficient. The implicit DNAT of return traffic 
in SNAT:ed connections is run in PREROUTE and should be before the bridge 
forwarding table..

Regards
Henrik

Re: snat bridge routes reply packets

[Amin Azez]

Henrik Nordstrom wrote:
> On Mon, 3 Oct 2005, Amin Azez wrote:
> 
>> Yes. And having saved the mac address (and physdev) it will use the
>> bridge output functions instead of the ip route functions.
> 
> 
> Saving the MAC should be sufficient. The implicit DNAT of return traffic
> in SNAT:ed connections is run in PREROUTE and should be before the
> bridge forwarding table..

Thanks for this tip. I've been working through the conntrack and natting
code again looking for the most judicious point to apply this.

There is a wiki up with my findings I'll announce in another post.

Sam