iptables / ebtables IP address intercept

iptables / ebtables IP address intercept

[Alex Bligh]

On a machine acting as a bridge, I want to intercept calls to a specific
IP address, and serve them locally. So, the bridge config looks like (e.g.)

  10.10.0.1/29     ethernet   eth0         tap1.1       10.10.0.2/29
  default router <-------------> br0 bridge <---------> testbox

                                    | 192.200.3.2/24
                                    | eth1
                                    V
                                second default router (192.200.3.1/24)


I have control over the bridge running br0, and I want to intercept
on the bridge tcp requests to (e.g.) 192.0.200.1:80, and remap them
locally to (e.g.) 192.0.200.2:8080. What I want to achieve is that
when testbox makes a connection to 192.0.200.1:80, this is remapped
by SNAT and DNAT within the bridge so that 192.0.200.2 sees a connection
to its port 8080 coming from 192.200.3.2 (i.e. the public IP of the
bridge).

I have a config which works if I put an IP address on the bridge's
ethernet interface of (e.g.) 10.10.0.3/29 (*)

The main components are:
  # masquerade on the output interface
  iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

  # allow backtrafic back in
  iptables -A FORWARD -i eth1 \! -o eth1 -m state \
    --state RELATED,ESTABLISHED -j ACCEPT

  # Rewrite destination address and port
  iptables -t nat -A PREROUTING --p tcp --match multiport --dports 80 \
    -d 192.0.200.1 -j DNAT --to 192.0.200.2:8080

This all works fine, but my problem is that I can't put an IP interface
on the ethernet interface (i.e. I can't do the step labeled (*) above)
because I don't know how the network is numbered (here it's labeled
10.10.0.0 but I don't know what the true labeling is).

So I am trying to use ebtables (which I am less familiar with) to turn
the bridge into a brouter. I use this:

 ebtables -t broute -A BROUTING -p IPv4 --ip-dst 192.0.200.1 \
  -j redirect

I think what that should do is redirect the MAC address for requests
to the IP address to the internal brouter, and indeed it seems to
work. My problem is on return traffic. The brouter unmangles the
IP addresses correctly, and wants to send traffic back to
10.10.0.2. However, it does this by looking in the internal routing
table, which results in the packet being sent out of eth1 (not
tap1.1) whereupon bad things happen. This is because the
machine doesn't (and can't) have a route to 10.10.0.0/24, so
doesn't know what interface to send it out of.

What I really want is for the masquerade line to remember not only
the input IP address but also the input interface associated with
the connection, and ignore the routing table. I am, however, open
to any other ideas. I should say that the bridge box concerned
carries hundreds of VLANs - I am happy using connmark to track
traffic, but I can't see how to match connmark in iptables and
use that to set output interface and output mac address.

-- 
Alex Bligh

Re: iptables / ebtables IP address intercept

[Grant Taylor]

On 07/31/10 09:24, Alex Bligh wrote:
> On a machine acting as a bridge, I want to intercept calls to a 
> specific IP address, and serve them locally. So, the bridge config 
> looks like (e.g.)

Ok...

> What I really want is for the masquerade line to remember not only 
> the input IP address but also the input interface associated with the 
> connection, and ignore the routing table. I am, however, open to any 
> other ideas. I should say that the bridge box concerned carries 
> hundreds of VLANs - I am happy using connmark to track traffic, but I 
> can't see how to match connmark in iptables and use that to set 
> output interface and output mac address.

It sounds like you want a layer 2 bridge (that is agnostic to the layer 
3 addressing that is passing through it) to be able to intercept traffic 
that is destined to one (or more) specific destination IP address(es). 
Correct?  (I'm presuming yes for the sake of conversation.)

I think I have a foundation for a rough idea but I'd like to run it past 
you to see if I'm going the correct direction or not.

Consider the following topology.

(R1)----[B1]----{C1}
          |
          |
         (R2)
          |
          |
         {S1}

R1 = existing default gateway
R2 = new default gateway
B1 = bridge
C1 = client
S1 = server

B1 bridges all three interfaces (called r1 / c1 / r2 respectively) together.

Normally all non-intercepted traffic flows through the r1 and c1 
interfaces of the bridge.  The intercepted traffic flows through the r2 
and c1 interfaces of the bridge.

To intercept traffic from C1 to S1, the bridge will match traffic that 
is destined to S1's IP address via R1's MAC address and DNAT it so that 
it goes to S1's IP address via R2's MAC address.  Thus the traffic will 
now flow to R2.  R2 will receive the traffic and route it normally.

This is where things get a bit interesting.  -  Either you will need to 
have R2 be in the same layer 3 network as C1 (easier) or you will need 
to do some interesting things at layer 2 to get the traffic across layer 
3 networks (harder but doable).

(For now I'm going to continue as if R2 is part of the same layer 3 
network as C1.  I think you can have R2 use the same IP address as R1 
with out conflict because B1 is separating the routers / preventing a 
conflict.)

S1 will receive the traffic and reply like normal.  S1's reply will need 
to go back to R2, which will route the traffic normally.  (If you don't 
want S1 to use R2 there are options, but we can get to those.)

B1 will match the reply traffic that is from S1's IP address via R2's 
MAC address and SNAT it so that it is from S1's IP address via R1's MAC 
address.  Thus the traffic will now flow to C1, there by establishing 
two way communications between its C1 and S1.

The two sticking points (which I believe can be worked around) that I 
see are that S1 needs to use R2 as it's default gateway and R2 needs to 
get traffic between multiple layer 3 networks.

You could probably have R2 SNAT / MASQUERADE traffic to S1 thus negating 
the need for S1 to use R2 as its default gateway.  But that pushes more 
complexity to R2 which I'd like to avoid for the moment.  (I say "for 
the moment" because I want to have a better understanding of how R2 can 
be configured.)

If R2 can be part of the same layer 3 network as C1, then R2 is simply a 
router that has multiple networks and can easily do the SNATing / 
MASQUERADEing.

If R2 can't be part of the same layer 3 network as C1, then you are 
going to have to do some interesting things to get the traffic to cross 
layer 3 networks.  Namely you are going to have to alter layer 3 IP 
addresses from layer 2.  ...  After reviewing the EBTables man page, you 
may have to invoke IPTables on layer 2 bridged traffic to alter the 
layer 3 IP addresses.  But, I believe that this can be done.

Seeing as how B1 is operating on layer 2 and R2 is operating on layer 3 
(clear separation of layers) it may be possible to combine them in to 
one bridging router (a.k.a. brouter).  Before I can say for sure, I need 
to know if what I have suggested will work for you or not.

Hopefully this gives you a direction to go.  At the very least let me 
know if I'm on the right track or not.



Grant. . . .

Re: iptables / ebtables IP address intercept

[Alex Bligh]

Grant,

--On 3 August 2010 11:54:14 -0500 Grant Taylor <gtaylor@riverviewtech.net> 
wrote:

>> What I really want is for the masquerade line to remember not only
>> the input IP address but also the input interface associated with the
>> connection, and ignore the routing table. I am, however, open to any
>> other ideas. I should say that the bridge box concerned carries
>> hundreds of VLANs - I am happy using connmark to track traffic, but I
>> can't see how to match connmark in iptables and use that to set
>> output interface and output mac address.
>
> It sounds like you want a layer 2 bridge (that is agnostic to the layer 3
> addressing that is passing through it) to be able to intercept traffic
> that is destined to one (or more) specific destination IP address(es).
> Correct?  (I'm presuming yes for the sake of conversation.)

Yes

> I think I have a foundation for a rough idea but I'd like to run it past
> you to see if I'm going the correct direction or not.
>
> Consider the following topology.
>
> (R1)----[B1]----{C1}
>           |
>           |
>          (R2)
>           |
>           |
>          {S1}
>
> R1 = existing default gateway
> R2 = new default gateway
> B1 = bridge
> C1 = client
> S1 = server
>
> B1 bridges all three interfaces (called r1 / c1 / r2 respectively)
> together.
>
> Normally all non-intercepted traffic flows through the r1 and c1
> interfaces of the bridge.  The intercepted traffic flows through the r2
> and c1 interfaces of the bridge.

Indeed, and (crucially) backtraffic flows the same way (except backwards).
R2 also does SNAT & DNAT as well as routing. Also B1 and R2 are
the same box (i.e. B1/R2 is a brouter)

Note it's intercepting traffic to Server X (somewhere behind R1), not
traffic to S1.

> To intercept traffic from C1 to S1, the bridge will match traffic that is
> destined to S1's IP address via R1's MAC address and DNAT it so that it
> goes to S1's IP address via R2's MAC address.  Thus the traffic will now
> flow to R2.  R2 will receive the traffic and route it normally.

Yes, that works fine.

> This is where things get a bit interesting.  -  Either you will need to
> have R2 be in the same layer 3 network as C1 (easier)

This needs to work irrespective of the L3 configuration of C1 and R1,
so that doesn't work.

> or you will need to
> do some interesting things at layer 2 to get the traffic across layer 3
> networks (harder but doable).

What I'm doing to get the outbound traffic working is the SNAT and DNAT
which is sufficient as B1/R2 are one box. However, the backtraffic
for SNAT assumes there is an L3 route on R2 to determine egress interface.
What I want is to match on (e.g.) CONNMARK and set the output interface
at the IPTABLES level. I can't do that, except by copying CONMARK into
MARK then using ip rules to policy route. I don't particularly want
to do that as it's not really very scalable.

[snip]

> B1 will match the reply traffic that is from S1's IP address via R2's MAC
> address and SNAT it so that it is from S1's IP address via R1's MAC
> address.  Thus the traffic will now flow to C1, there by establishing two
> way communications between its C1 and S1.

This fails because though S1's IP address is restored by the SNAT of the
backtraffic, the default route for S1's IP address is out some other
interface (normally B1/R2's default route).

> The two sticking points (which I believe can be worked around) that I see
> are that S1 needs to use R2 as it's default gateway

That is not necessary. SNAT means the outbound packets (inbound to S1)
contain B1/R2's source IP address, so provided S1 can reach that (which is
by assumption true as S1 is elsewhere on the internet and B1/R2 has a
public IP or something that passes for one), backtraffic from S1 contains
B1/R2's IP address as a destination, and it thus reaches B1/R2 subsequently.

> and R2 needs to get
> traffic between multiple layer 3 networks.

Not quite sure what that means as that's what a router does by definition!

> You could probably have R2 SNAT / MASQUERADE traffic to S1 thus negating
> the need for S1 to use R2 as its default gateway.

Yes, that's what I'm doing.

> But that pushes more
> complexity to R2 which I'd like to avoid for the moment.  (I say "for the
> moment" because I want to have a better understanding of how R2 can be
> configured.)

Sadly it's necessary.

> If R2 can be part of the same layer 3 network as C1, then R2 is simply a
> router that has multiple networks and can easily do the SNATing /
> MASQUERADEing.
>
> If R2 can't be part of the same layer 3 network as C1, then you are going
> to have to do some interesting things to get the traffic to cross layer 3
> networks.  Namely you are going to have to alter layer 3 IP addresses
> from layer 2.  ...  After reviewing the EBTables man page, you may have
> to invoke IPTables on layer 2 bridged traffic to alter the layer 3 IP
> addresses.  But, I believe that this can be done.

The step I am having difficulty with is after the SNAT/DNAT has been
undone on reverse traffic, it needs to know which interface (which may be
a bridge) to push the traffic out of. I don't think ebtables is much
use here because it hits the routing step after SNAT/DNAT which causes
the problem, and the backtraffic never hits a bridge.

> Seeing as how B1 is operating on layer 2 and R2 is operating on layer 3
> (clear separation of layers) it may be possible to combine them in to one
> bridging router (a.k.a. brouter).  Before I can say for sure, I need to
> know if what I have suggested will work for you or not.

That's what I am doing :-)

> Hopefully this gives you a direction to go.  At the very least let me
> know if I'm on the right track or not.

You are on the right track, but sadly how to get this working with
iptables still eludes me.

I know it's possible to do programmatically as I have knocked up some
raw socket stuff which is working in userspace, which does all the NAT
steps and MAC manipulation by hand (that was fun). I'd just rather not
do it that way.

-- 
Alex Bligh

Re: iptables / ebtables IP address intercept

[Grant Taylor]

On 08/03/10 13:06, Alex Bligh wrote:
> Indeed, and (crucially) backtraffic flows the same way (except 
> backwards). R2 also does SNAT & DNAT as well as routing. Also B1 and 
> R2 are the same box (i.e. B1/R2 is a brouter)

*nod*

> Note it's intercepting traffic to Server X (somewhere behind R1), not 
> traffic to S1.

OK.

Either way, B1 is intercepting traffic and re-routing it elsewhere.

Are you saying that the new destination of the traffic is also out R1?

I.e. the following topology?

{S1}--- BBI ---(R1)---[B1]---{C1}

Where BBI is the BigBadInternet

> This needs to work irrespective of the L3 configuration of C1 and R1, 
> so that doesn't work.

Ok.

That means that the NATing is going to be ""fun.

> What I'm doing to get the outbound traffic working is the SNAT and 
> DNAT which is sufficient as B1/R2 are one box. However, the 
> backtraffic for SNAT assumes there is an L3 route on R2 to determine 
> egress interface.

Agreed.

> What I want is to match on (e.g.) CONNMARK and set 
> the output interface at the IPTABLES level. I can't do that, except 
> by copying CONMARK into MARK then using ip rules to policy route. I 
> don't particularly want to do that as it's not really very scalable.

I'm thinking that IPTables' Bridge NetFilter option will allow us to do 
something else.

> This fails because though S1's IP address is restored by the SNAT of 
> the backtraffic, the default route for S1's IP address is out some 
> other interface (normally B1/R2's default route).

*nod*

> That is not necessary. SNAT means the outbound packets (inbound to 
> S1) contain B1/R2's source IP address, so provided S1 can reach that 
> (which is by assumption true as S1 is elsewhere on the internet and 
> B1/R2 has a public IP or something that passes for one), backtraffic 
> from S1 contains B1/R2's IP address as a destination, and it thus 
> reaches B1/R2 subsequently.

Agreed.

>> R2 needs to get traffic between multiple layer 3 networks.
> 
> Not quite sure what that means as that's what a router does by 
> definition!

Sorry.  Re-reading what I wrote, I see that I wasn't as clear as I 
needed to be.

I was thinking that a device that was not in one (or both) of the layer 
3 networks would need to alter the layer 3 traffic so end systems could 
communicate.

More specifically, using a layer 2 device to alter the layer 3 IP 
addresses, and possibly the layer 2 MAC addresses.  (This is where the 
IPTables' Bridge NetFilter option comes in to play.)

> Yes, that's what I'm doing.

*nod*

> Sadly it's necessary.

Ok.

This complicates things.

(Time to do some thinking about IPTables' Bridge NetFilter options.)

> The step I am having difficulty with is after the SNAT/DNAT has been 
> undone on reverse traffic, it needs to know which interface (which 
> may be a bridge) to push the traffic out of.

Here in lies your problem.

You are going to have to have a route (of some sort) to get back from B1 
to C1.

Now /what/ said route is, doesn't matter.  Which is a good thing.

> I don't think ebtables 
> is much use here because it hits the routing step after SNAT/DNAT 
> which causes the problem, and the backtraffic never hits a bridge.

I don't know if EBTables its self is going to play that much (more) of a 
role in your configuration.  That being said, bridging is going to 
continue to play a considerable role.

> That's what I am doing :-)

*nod*

(Again, most of the differences between B1 and R2 were to differentiate 
between the layers in my head as I was figuring things out.)

 From this point on, I'm going to refer to B1 and R2 collectively as BR1.

> You are on the right track, but sadly how to get this working with 
> iptables still eludes me.

Now the fun begins.  (Now we are getting in to something that I haven't 
messed with in a very long time.)

{S1}--- BBI ---(R1)---[BR1]---{C1}

Here's my idea is to bridge the r1 and c1 interfaces together in to the 
br1 bridge and route modified packets.  The key being the "modified 
packets".

It is (or was last time I needed to do it years ago) to use IPTables' 
Bridge NetFilter option to allow IPTables (a layer 3 toolbox) to operate 
on bridged (a layer 2 operation) packets before they entered the logical 
bridge interface (as seen by the routing code).  Thus we could modify 
packets that came in to the bridge's the physical c1 interface before 
they went out the bridge's logical br1 interface on up to the routing 
code.  (Here in lies the magic.)  -  We use IPTables' Bridge NetFilter 
option to NAT the unknown layer 3 IP addresses in to a known IP address 
for routing to work with.  Conceptually it would function like this:

  - Bridge / EBTables* receives / intercepts the IP packet.
       C1 -> S1
  - Bridge NetFilter code modifies the IP packet.
       C1' -> S1
  - IPTables DNATs/SNATs the packet.
       BR1 -> S1
  - Kernel sends the packet out to S1.
       BR1 -> S1

Similarly the reply traffic would take the reverse process.

  - Kernel receives the IP packet.
       S1 -> BR1
  - IPTables unSNATs / unDNATs the IP packet.
       S1 -> C1'
  - Bridge NetFilter code unmodifies the IP packet.
       S1 -> C1
  - Bridge sends the IP packet out to C1.
       S1 -> C1

I will need to do some more thinking on how to utilize IPTables's Bridge 
NetFilter code / rules to do what is needed to be done.  Give me a bit 
and I'll re-reply.

> I know it's possible to do programmatically as I have knocked up some 
> raw socket stuff which is working in userspace, which does all the 
> NAT steps and MAC manipulation by hand (that was fun). I'd just 
> rather not do it that way.

I don't think you are going to need to do it progromatically.

As I'm brushing up on kernel options, you may want to consider the raw 
table so that you can use the NOTRACK targets so that you don't consume 
resources tracking all the packets that aren't being intercepted.



Grant. . . .

Re: iptables / ebtables IP address intercept

[Alex Bligh]

Grant,

--On 3 August 2010 15:23:49 -0500 Grant Taylor <gtaylor@riverviewtech.net> 
wrote:

>
>> Note it's intercepting traffic to Server X (somewhere behind R1), not
>> traffic to S1.
>
> OK.
>
> Either way, B1 is intercepting traffic and re-routing it elsewhere.
>
> Are you saying that the new destination of the traffic is also out R1?
>
> I.e. the following topology?
>
> {S1}--- BBI ---(R1)---[B1]---{C1}
>
> Where BBI is the BigBadInternet

No:


(R1)----[B1]----{C1}
          |
          |
         (R2)
          |
         BBI
          |
         {S1}

R1 might or might not be connected to BBI, or R1/C1 might be in 1918
space

>>> R2 needs to get traffic between multiple layer 3 networks.
>>
>> Not quite sure what that means as that's what a router does by
>> definition!
>
> Sorry.  Re-reading what I wrote, I see that I wasn't as clear as I needed
> to be.
>
> I was thinking that a device that was not in one (or both) of the layer 3
> networks would need to alter the layer 3 traffic so end systems could
> communicate.
>
> More specifically, using a layer 2 device to alter the layer 3 IP
> addresses, and possibly the layer 2 MAC addresses.  (This is where the
> IPTables' Bridge NetFilter option comes in to play.)

OK - that looks promising. I'd only investigated iptables & ebtables.

>> You are on the right track, but sadly how to get this working with
>> iptables still eludes me.
>
> Now the fun begins.  (Now we are getting in to something that I haven't
> messed with in a very long time.)
>
> {S1}--- BBI ---(R1)---[BR1]---{C1}

Note BBI hangs of BR1 not R1 - I think that makes things easier.

> Here's my idea is to bridge the r1 and c1 interfaces together in to the
> br1 bridge and route modified packets.  The key being the "modified
> packets".
>
> It is (or was last time I needed to do it years ago) to use IPTables'
> Bridge NetFilter option to allow IPTables (a layer 3 toolbox) to operate
> on bridged (a layer 2 operation) packets before they entered the logical
> bridge interface (as seen by the routing code).  Thus we could modify
> packets that came in to the bridge's the physical c1 interface before
> they went out the bridge's logical br1 interface on up to the routing
> code.  (Here in lies the magic.)  -  We use IPTables' Bridge NetFilter
> option to NAT the unknown layer 3 IP addresses in to a known IP address
> for routing to work with.  Conceptually it would function like this:
>
>   - Bridge / EBTables* receives / intercepts the IP packet.
>        C1 -> S1
>   - Bridge NetFilter code modifies the IP packet.
>        C1' -> S1
>   - IPTables DNATs/SNATs the packet.
>        BR1 -> S1
>   - Kernel sends the packet out to S1.
>        BR1 -> S1
>
> Similarly the reply traffic would take the reverse process.
>
>   - Kernel receives the IP packet.
>        S1 -> BR1
>   - IPTables unSNATs / unDNATs the IP packet.
>        S1 -> C1'

I presume there is a route step here, but C1' is designed to be on the
correct interface.

>   - Bridge NetFilter code unmodifies the IP packet.
>        S1 -> C1
>   - Bridge sends the IP packet out to C1.
>        S1 -> C1
>
> I will need to do some more thinking on how to utilize IPTables's Bridge
> NetFilter code / rules to do what is needed to be done.  Give me a bit
> and I'll re-reply.

Sure - thanks.

>> I know it's possible to do programmatically as I have knocked up some
>> raw socket stuff which is working in userspace, which does all the
>> NAT steps and MAC manipulation by hand (that was fun). I'd just
>> rather not do it that way.
>
> I don't think you are going to need to do it progromatically.
>
> As I'm brushing up on kernel options, you may want to consider the raw
> table so that you can use the NOTRACK targets so that you don't consume
> resources tracking all the packets that aren't being intercepted.

That's actually quite simple, as because BBI is not behind R1, all the
non-intercepted traffic is simply bridged.

-- 
Alex Bligh

Re: iptables / ebtables IP address intercept

[Grant Taylor]

On 08/04/10 03:08, Alex Bligh wrote:
> R1 might or might not be connected to BBI, or R1/C1 might be in 1918 
> space

Ok.

> OK - that looks promising. I'd only investigated iptables & ebtables.

*nod*

> Note BBI hangs of BR1 not R1 - I think that makes things easier.

Seeing as how BR1 is a bridge and router, it doesn't make that much 
difference.

> I presume there is a route step here, but C1' is designed to be on 
> the correct interface.

Yes.

The reply traffic will be routed at layer 3, and then at layer 2 will be 
altered from the one layer 3 (that was routed) to the proper layer 3 
(that isn't routed).

> Sure - thanks.

I've done some more thinking and I strongly believe that what needs to 
be done can be, I just don't have a way to test this at the moment.  (I 
will see if I have some time this weekend to throw together a VM.)

> That's actually quite simple, as because BBI is not behind R1, all 
> the non-intercepted traffic is simply bridged.

*nod*



Grant. . . .