Filtering on bridges

Filtering on bridges

[Steve Hill]

I have previously used iptables to filter traffic on bridged interfaces 
using the physdev module.  However, recently it seems there was a change 
in the semantics of physdev:

For bridged traffic (i.e. traffic that is coming in through one physical 
NIC, traversing a bridge and being sent out of another one without being 
routed), physdev still works as expected.

However, for traffic that has gone through the machine's own IP stack 
(either by being routed or by being generated locally), --physdev-out is 
no longer allowed.  At the time the iptables rules are being executed, 
the only thing you know is the logical bridge interface it is being 
routed to rather than the physical NIC it will eventually be sent from. 
  Is there a recommended method of filtering this traffic based on the 
physical NIC it is being sent out of, now that the deferred rule 
functionality has been removed?  ebtables doesn't really seem to be an 
option since it is nowhere near as powerful as iptables when it comes to 
IP filtering.

Background:
I'm running virtualised servers which are bridged to the physical 
network (this makes VM migration between physical hosts possible - doing 
this using a routed infrastructure would be messy since the routers 
themselves would need to be adjusted during VM migration).  I run 
iptables/ip6tables on the host machine in order to firewall the VMs and 
also for statistics reporting - these iptables rules reference each VM's 
network interface.  I would like to be able to filter routed traffic in 
the same way as the bridged traffic, but this involves knowing which VM 
it is destined for (and hence which NIC it will be sent to).

-- 

  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve@opendium.com
    Email:            steve@opendium.com
    Phone:            sip:steve@opendium.com

Sales / enquiries contacts:
    Email:            sales@opendium.com
    Phone:            +44-844-9791439 / sip:sales@opendium.com

Support contacts:
    Email:            support@opendium.com
    Phone:            +44-844-4844916 / sip:support@opendium.com

Re: Filtering on bridges

[Jan Engelhardt]

On Wednesday 2011-12-21 11:16, Steve Hill wrote:

>For bridged traffic (i.e. traffic that is coming in through one physical NIC,
>traversing a bridge and being sent out of another one without being routed),
>physdev still works as expected.
>
>However, for traffic that has gone through the machine's own IP stack (either
>by being routed or by being generated locally), --physdev-out is no longer
>allowed.  At the time the iptables rules are being executed, the only thing you
>know is the logical bridge interface it is being routed to rather than the
>physical NIC it will eventually be sent from.  Is there a recommended method of
>filtering this traffic based on the physical NIC it is being sent out of, now
>that the deferred rule functionality has been removed?  ebtables doesn't really
>seem to be an option since it is nowhere near as powerful as iptables when it
>comes to IP filtering.

Mark the packets leaving on brX with iptables, then use ebtables to check
for the physical interface plus the mark.

Re: Filtering on bridges

[Steve Hill]

On 21/12/11 13:30, Jan Engelhardt wrote:

> Mark the packets leaving on brX with iptables, then use ebtables to check
> for the physical interface plus the mark.

At mark-time I wouldn't actually know where the packet is going, so 
wouldn't know what criteria to mark based on...

-- 

  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve@opendium.com
    Email:            steve@opendium.com
    Phone:            sip:steve@opendium.com

Sales / enquiries contacts:
    Email:            sales@opendium.com
    Phone:            +44-844-9791439 / sip:sales@opendium.com

Support contacts:
    Email:            support@opendium.com
    Phone:            +44-844-4844916 / sip:support@opendium.com

Re: Filtering on bridges

[Niccolò Belli]

You *can't*, that's simple.
Forget the ebtables/marking shit, I did a firewall with it (which worked 
flawlessly of course) and then I trashed it after seeing how disgusting 
was the code. Do not use linux, it simply can't do the task in a 
feasible way. We are really gone backwards concerning bridging firewalls.

Niccolò

Re: Filtering on bridges

[Jan Engelhardt]

On Wednesday 2011-12-21 14:48, Steve Hill wrote:

> On 21/12/11 13:30, Jan Engelhardt wrote:
>
>> Mark the packets leaving on brX with iptables, then use ebtables to check
>> for the physical interface plus the mark.
>
>At mark-time I wouldn't actually know where the packet is going, so wouldn't
>know what criteria to mark based on...

Yes you do know where it is going; the route is set on FORWARD/OUTPUT 
(see nf-packet-flow.svg), therefore, you can mark, and as for the bridge 
phys interface, it too will be set at one point, at which time you can 
check the mark.

Re: Filtering on bridges

[Steve Hill]

On 21/12/11 18:36, Jan Engelhardt wrote:

> Yes you do know where it is going; the route is set on FORWARD/OUTPUT
> (see nf-packet-flow.svg)

I only know it is going to the bridge.  I don't know which member of 
that bridge it will eventually be sent out of.  In short: I don't know 
where it is going (in enough detail to be useful).

-- 

  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve@opendium.com
    Email:            steve@opendium.com
    Phone:            sip:steve@opendium.com

Sales / enquiries contacts:
    Email:            sales@opendium.com
    Phone:            +44-844-9791439 / sip:sales@opendium.com

Support contacts:
    Email:            support@opendium.com
    Phone:            +44-844-4844916 / sip:support@opendium.com

Re: Filtering on bridges

[Jan Engelhardt]

On Thursday 2011-12-22 00:21, Steve Hill wrote:

> On 21/12/11 18:36, Jan Engelhardt wrote:
>
>>Yes you do know where it is going; the route is set on FORWARD/OUTPUT
>>(see nf-packet-flow.svg)
>
>I only know it is going to the bridge.  I don't know which member of 
>that bridge it will eventually be sent out of.  In short: I don't know 
>where it is going (in enough detail to be useful).

But the kernel knows, and that's really all that is needed.

Re: Filtering on bridges

[Steve Hill]

On 21/12/11 23:31, Jan Engelhardt wrote:

>>> Yes you do know where it is going; the route is set on FORWARD/OUTPUT
>>> (see nf-packet-flow.svg)
>>
>> I only know it is going to the bridge.  I don't know which member of
>> that bridge it will eventually be sent out of.  In short: I don't know
>> where it is going (in enough detail to be useful).
>
> But the kernel knows, and that's really all that is needed.

I think we have a misunderstanding somewhere:

I need to filter traffic from the local IP stack (whether that be 
locally generated or routed - lets concentrate on the locally generated 
traffic for now), and which set of iptables rules apply is to be based 
on which physical NIC the traffic will egress.

In the iptables/ip6tables OUTPUT chain, the -o option matches against 
which bridge the traffic is going to (not the physical NIC).  However, 
this is not useful because I'm interested in the physical interface, not 
the bridge interface.  I used to be able to use --physdev-out to match 
the physical NIC, but this is no longer allowed in OUTPUT.

Looking at http://jengelh.medozas.de/images/nf-packet-flow.svg, it could 
probably use some arrows to help understand the direction that some of 
the links should be followed in (the direction of the link between 
"routing decision" and "bridging decision" seems ambiguous to me). 
However, I can't see any "bridging decision" for locally generated 
traffic, so I'm not sure at what point the physical output NIC is 
determined.  I _presume_ that the bridging decision is made between 
"iptables-nat-postrouting" and "ebtables-nat-output", in which case 
there does indeed seem to be no way to match the physical output 
interface from iptables, since no iptables rules are invoked after that 
point.  I guess what is really needed is an extra iptables filter chain 
between "ebtables-nat-postrouting" and "egress".

So at the moment, the only way I can think of doing the filtering is to 
allow the packet to run through *all* the iptables rules without 
matching the physical output NIC and set one bit of the fwmark for each 
physical interface I would let the packet egress.  Then in ebtables 
(where we know the physical interface) filter the packets by looking at 
the fwmark bit that I've used to indicate that interface.  This method 
is pretty unscalable (fwmark is 32 bits long, so we're limited to 32 
interfaces, minus any other things that the fwmark will be used for, 
such as QoS) and also will be absolute hell to manage.

So any other good ideas for how to accomplish this filtering?  (I 
understand the reasons for removing the physdev-out support for 
unbridged traffic, but why on earth hasn't it been replaced with 
something to do a similar job, such as an additional iptables chain 
after the bridging decision rather than wholesale removing a useful 
feature?)

-- 

  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve@opendium.com
    Email:            steve@opendium.com
    Phone:            sip:steve@opendium.com

Sales / enquiries contacts:
    Email:            sales@opendium.com
    Phone:            +44-844-9791439 / sip:sales@opendium.com

Support contacts:
    Email:            support@opendium.com
    Phone:            +44-844-4844916 / sip:support@opendium.com

Re: Filtering on bridges

[Jan Engelhardt]

On Thursday 2011-12-22 11:56, Steve Hill wrote:
>>
>> But the kernel knows, and that's really all that is needed.
>
>I think we have a misunderstanding somewhere:
>[...]

>Looking at nf-packet-flow.svg, [...] the direction of the link
>between "routing decision" and "bridging decision" seems ambiguous
>to me).

In that graphic, when there is a choice, the network stack will pick
only one.

>[..]
>So at the moment, the only way I can think of doing the filtering is to allow
>the packet to run through *all* the iptables rules without matching the
>physical output NIC and set one bit of the fwmark for each physical interface I
>would let the packet egress.  Then in ebtables (where we know the physical
>interface) filter the packets by looking at the fwmark bit that I've used to
>indicate that interface. This method is pretty unscalable (fwmark is 32
>bits)

As for filtering, which I had gathered was what you wanted, you could 
set the fwmark to indicate drop-or-not-drop (rather than a bit for each 
interface). As for QoS though, indeed there is a little issue.

>So any other good ideas for how to accomplish this filtering?  (I 
>understand the reasons for removing the physdev-out support for 
>unbridged traffic, but why on earth hasn't it been replaced with 
>something to do a similar job, such as an additional iptables chain 
>after the bridging decision rather than wholesale removing a useful 
>feature?)

Checking the source for possibilities..

Re: Filtering on bridges

[Steve Hill]

On 22/12/11 16:28, Jan Engelhardt wrote:

>> So at the moment, the only way I can think of doing the filtering is to allow
>> the packet to run through *all* the iptables rules without matching the
>> physical output NIC and set one bit of the fwmark for each physical interface I
>> would let the packet egress.  Then in ebtables (where we know the physical
>> interface) filter the packets by looking at the fwmark bit that I've used to
>> indicate that interface. This method is pretty unscalable (fwmark is 32
>> bits)
>
> As for filtering, which I had gathered was what you wanted, you could
> set the fwmark to indicate drop-or-not-drop (rather than a bit for each
> interface).

Nope, can't do that - the iptables rules aren't going to know whether 
the packet needs to be dropped or not since it doesn't know which 
physical NIC it will egress - each NIC has its own (different) set of 
filtering rules, so without knowing the NIC, iptables won't know which 
set of filtering rules to apply and therefore whether the packet is to 
be dropped or not.

-- 

  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve@opendium.com
    Email:            steve@opendium.com
    Phone:            sip:steve@opendium.com

Sales / enquiries contacts:
    Email:            sales@opendium.com
    Phone:            +44-844-9791439 / sip:sales@opendium.com

Support contacts:
    Email:            support@opendium.com
    Phone:            +44-844-4844916 / sip:support@opendium.com

Re: Filtering on bridges

[Jan Engelhardt]

On Thursday 2011-12-22 18:36, Steve Hill wrote:

> On 22/12/11 16:28, Jan Engelhardt wrote:
>
>>>So at the moment, the only way I can think of doing the filtering
>>>is to allow the packet to run through *all* the iptables rules
>>>without matching the physical output NIC and set one bit of the
>>>fwmark for each physical interface I would let the packet egress.
>>>Then in ebtables (where we know the physical interface) filter the
>>>packets by looking at the fwmark bit that I've used to indicate
>>>that interface. This method is pretty unscalable (fwmark is 32
>>>bits)
>>
>>As for filtering, which I had gathered was what you wanted, you
>>could set the fwmark to indicate drop-or-not-drop (rather than a
>>bit for each interface).
>
>Nope, can't do that - the iptables rules aren't going to know
>whether the packet needs to be dropped or not since it doesn't know
>which physical NIC it will egress

What I mean is that with the mark, you record whether this is a
potential candidate for dropping. E.g. if

  tcp 22 eth0 -> drop, tcp 22 eth1 -> accept

you could

  -A OUTPUT -o br0 -p tcp --dport 22 -j MARK --set-mark [ssh-candidate-bit]
  ebtables -m mark --mark ssh-candidate-bit/ssh-candidate-bit -o [eth0/eth1] -j [DROP/ACCEPT]...

Re: Filtering on bridges

[Vigneswaran R]

On Friday 23 December 2011 03:35 AM, Jan Engelhardt wrote:
>
> On Thursday 2011-12-22 18:36, Steve Hill wrote:
>
>> On 22/12/11 16:28, Jan Engelhardt wrote:
>>
>>>> So at the moment, the only way I can think of doing the filtering
>>>> is to allow the packet to run through *all* the iptables rules
>>>> without matching the physical output NIC and set one bit of the
>>>> fwmark for each physical interface I would let the packet egress.
>>>> Then in ebtables (where we know the physical interface) filter the
>>>> packets by looking at the fwmark bit that I've used to indicate
>>>> that interface. This method is pretty unscalable (fwmark is 32
>>>> bits)
>>>
>>> As for filtering, which I had gathered was what you wanted, you
>>> could set the fwmark to indicate drop-or-not-drop (rather than a
>>> bit for each interface).
>>
>> Nope, can't do that - the iptables rules aren't going to know
>> whether the packet needs to be dropped or not since it doesn't know
>> which physical NIC it will egress

Sorry for interrupting your discussion. I am following this thread from 
the beginning. However, I couldn't get exactly how your setup looks 
like. If possible, could you please give a simple (ascii) pictorial 
representation of your setup. This may help more people (normal iptable 
users like me) to understand the discussion better. Thank you.

Regards,
Vignesh


>
> What I mean is that with the mark, you record whether this is a
> potential candidate for dropping. E.g. if
>
>    tcp 22 eth0 ->  drop, tcp 22 eth1 ->  accept
>
> you could
>
>    -A OUTPUT -o br0 -p tcp --dport 22 -j MARK --set-mark [ssh-candidate-bit]
>    ebtables -m mark --mark ssh-candidate-bit/ssh-candidate-bit -o [eth0/eth1] -j [DROP/ACCEPT]...

Re: Filtering on bridges

[Steve Hill]

On 23/12/11 04:33, Vigneswaran R wrote:

> Sorry for interrupting your discussion. I am following this thread from
> the beginning. However, I couldn't get exactly how your setup looks
> like. If possible, could you please give a simple (ascii) pictorial
> representation of your setup. This may help more people (normal iptable
> users like me) to understand the discussion better. Thank you.

+-----------------------------------------------------------+
|                        Linux Server                       |
|                                                           |
|                            (br0)                          |
|                              |                            |
|     +------------+-----------+-----------+-----------+    |
|     |            |           |           |           |    |
|  (eth0)       (eth1)      (eth2)      (eth3)      (eth4)  |
+-----------------------------------------------------------+
       |            |           |           |           |
       |            |           |           |           |
+----------+  +--------+  +--------+  +--------+  +--------+
| Upstream |  | Other  |  | Other  |  | Other  |  | Other  |
|  Switch  |  | Server |  | Server |  | Server |  | Server |
+----------+  +--------+  +--------+  +--------+  +--------+


As you can see, the Linux server has several network interfaces, all of 
which are bridged together.  I'm using iptables to filter the traffic 
between those NICs.  Filtering bridged traffic largely works as expected 
(although there is an issue that using REJECT filter rules causes the 
kernel to panic as the ICMP response is injected into the bridge!)  This 
means I can write iptables filter rules in the FORWARD chain using the 
physdev module to reference the physical NICs (e.g. "DROP TCP traffic 
from eth0 to eth1 that has a destination port == 25")

The issue I'm having is that traffic that has passed through the local 
IP stack (whether that be from a local process, or routed) is handled 
differently to traffic that is being bridged - iptables is no longer 
aware of the physical NIC from which the traffic will egress.  I.e. for 
this traffic I can only create rules referencing the whole bridge (br0) 
rather than individual physical NICs (eth0, eth1, etc.).


The following is NOT a description of what actually happens, it is what 
I believe would be a cleaner model than the current system:
Logically, I would say that routing and bridging are two completely 
different processes, so should have separate chains.  i.e. the 
PREROUTING, FORWARD, POSTROUTING and OUTPUT chains should refer *only* 
to traffic that is passing through the IP stack, and there should be 
separate PREBRIDGE, FORWARDBRIDGE, POSTBRIDGE chains for the bridged 
traffic.  The bridge interface (br0) should be seen as just another 
member of the bridge, exactly like the physical NICs.

So following the above idea, a packet generated by a local process would 
pass though:
iptables:OUTPUT (egress NIC: br0)
iptables:POSTROUTING (egress NIC: br0)
ebtables:BROUTING (ingress NIC: br0)
ebtables:PREROUTING (ingress NIC: br0)
iptables:PREBRIDGE (ingress NIC: br0)
ebtables:FORWARD (ingress NIC: br0, egress NIC: eth0)
iptables:FORWARDBRIDGE (ingress NIC: br0, egress NIC: eth0)
iptables:POSTBRIDGE (ingress NIC: br0, egress NIC: eth0)
ebtables:POSTROUTING (ingress NIC: br0, egress NIC: eth0)

And similarly, a bridged packet would look something like:
ebtables:BROUTING (ingress NIC: eth1)
ebtables:PREROUTING (ingress NIC: eth1)
iptables:PREBRIDGE (ingress NIC: eth1)
ebtables:FORWARD (ingress NIC: eth1, egress NIC: eth0)
iptables:FORWARDBRIDGE (ingress NIC: eth1, egress NIC: eth0)
iptables:POSTBRIDGE (ingress NIC: eth1, egress NIC: eth0)
ebtables:POSTROUTING (ingress NIC: eth1, egress NIC: eth0)

Unfortunately this isn't what happens :)



Hope that explains what I'm talking about anyway.  In my case, the 
"other servers" are actually virtual machines running on the Linux 
server, so "eth1..4" aren't really physical NICs, they are tun/tap 
interfaces.  But for the purpose of this discussion, that is unimportant.

-- 

  - Steve Hill
    Technical Director
    Opendium Limited     http://www.opendium.com

Direct contacts:
    Instant messager: xmpp:steve@opendium.com
    Email:            steve@opendium.com
    Phone:            sip:steve@opendium.com

Sales / enquiries contacts:
    Email:            sales@opendium.com
    Phone:            +44-844-9791439 / sip:sales@opendium.com

Support contacts:
    Email:            support@opendium.com
    Phone:            +44-844-4844916 / sip:support@opendium.com

Re: Filtering on bridges

[Jan Engelhardt]

On Tuesday 2012-01-03 14:15, Steve Hill wrote:
>
>The following is NOT a description of what actually happens, it is
>what I believe would be a cleaner model than the current system:
>Logically, I would say that routing and bridging are two completely
>different processes, so should have separate chains. i.e. the
>PREROUTING, FORWARD, POSTROUTING and OUTPUT chains should refer
>*only* to traffic that is passing through the IP stack, and there
>should be separate PREBRIDGE, FORWARDBRIDGE, POSTBRIDGE chains for
>the bridged traffic.

The packet *is* being passed through the IP stack - if you don't want
that, set net.bridge.bridge-nf-call-{arptables,ip6tables,iptables}=0.

>The bridge interface (br0) should be seen as just another member of
>the bridge, exactly like the physical NICs.

I don't see the point of this other than confusing the user.
After all, it says everywhere that bridges cannot be members
of bridges.

>So following the above idea, a packet generated by a local process
>would pass though:
>
>iptables:OUTPUT (egress NIC: br0)
>iptables:POSTROUTING (egress NIC: br0)
>ebtables:BROUTING (ingress NIC: br0)
>ebtables:PREROUTING (ingress NIC: br0)

I know there is a saying "after the game is before the game",
but you can't apply that to Linux here.

>iptables:PREBRIDGE (ingress NIC: br0)
>ebtables:FORWARD (ingress NIC: br0, egress NIC: eth0)
>iptables:FORWARDBRIDGE (ingress NIC: br0, egress NIC: eth0)
>iptables:POSTBRIDGE (ingress NIC: br0, egress NIC: eth0)
>ebtables:POSTROUTING (ingress NIC: br0, egress NIC: eth0)

FORWARD and PRE*-logicry is never invoked in the local-output path
by design, simply because there is nothing forwarded here.

> And similarly, a bridged packet would look something like:
> ebtables:BROUTING (ingress NIC: eth1)
> ebtables:PREROUTING (ingress NIC: eth1)
> iptables:PREBRIDGE (ingress NIC: eth1)
> ebtables:FORWARD (ingress NIC: eth1, egress NIC: eth0)
> iptables:FORWARDBRIDGE (ingress NIC: eth1, egress NIC: eth0)
> iptables:POSTBRIDGE (ingress NIC: eth1, egress NIC: eth0)
> ebtables:POSTROUTING (ingress NIC: eth1, egress NIC: eth0)
>
> Unfortunately this isn't what happens :)

The wording PREBRIDGE does not make sense, since the IP level does
not, by abstract fashion, know or care about any bridge. This is why
the IP-level hooks are called xxxROUTING, and skb->dev is set to
"br0", necessiting a special -m physdev "layer-breaker" to get at
the real dev.

If anything, the ebtables hooks {BROUTING, PREROUTING, FORWARD,
OUTPUT, POSTROUTING} would have been called something like {BROUTING,
PREBRIDGE, BRIDGED, OUTPUT, POSTBRIDGE} -- because that it the bridge
terminology set --, but alas, the initial developer did not choose
so.

Re: Filtering on bridges

[John A. Sullivan III]

On Tue, 2012-01-03 at 13:15 +0000, Steve Hill wrote:
> On 23/12/11 04:33, Vigneswaran R wrote:
> 
> > Sorry for interrupting your discussion. I am following this thread from
> > the beginning. However, I couldn't get exactly how your setup looks
> > like. If possible, could you please give a simple (ascii) pictorial
> > representation of your setup. This may help more people (normal iptable
> > users like me) to understand the discussion better. Thank you.
> 
> +-----------------------------------------------------------+
> |                        Linux Server                       |
> |                                                           |
> |                            (br0)                          |
> |                              |                            |
> |     +------------+-----------+-----------+-----------+    |
> |     |            |           |           |           |    |
> |  (eth0)       (eth1)      (eth2)      (eth3)      (eth4)  |
> +-----------------------------------------------------------+
>        |            |           |           |           |
>        |            |           |           |           |
> +----------+  +--------+  +--------+  +--------+  +--------+
> | Upstream |  | Other  |  | Other  |  | Other  |  | Other  |
> |  Switch  |  | Server |  | Server |  | Server |  | Server |
> +----------+  +--------+  +--------+  +--------+  +--------+
> 
> 
> As you can see, the Linux server has several network interfaces, all of 
> which are bridged together.  I'm using iptables to filter the traffic 
> between those NICs.  Filtering bridged traffic largely works as expected 
> (although there is an issue that using REJECT filter rules causes the 
> kernel to panic as the ICMP response is injected into the bridge!)  This 
> means I can write iptables filter rules in the FORWARD chain using the 
> physdev module to reference the physical NICs (e.g. "DROP TCP traffic 
> from eth0 to eth1 that has a destination port == 25")
> 
> The issue I'm having is that traffic that has passed through the local 
> IP stack (whether that be from a local process, or routed) is handled 
> differently to traffic that is being bridged - iptables is no longer 
> aware of the physical NIC from which the traffic will egress.  I.e. for 
> this traffic I can only create rules referencing the whole bridge (br0) 
> rather than individual physical NICs (eth0, eth1, etc.).
<snip>
I wonder if you could integrate tc and iptables here.  I haven't thought
it all the way through but you could do something like setup an ingress
qdisc and use a tc filter ipt action (action ipt -j MARK --set-mark) to
mark packets based upon the iif so you know where it came and then
examine that mark on egress with a u32 filter with all the other match
parameters and drop it if necessary.  I'm not all that familiar with
using tc is such a way but that would get you back to the physical
interfaces - John