ebtables and iptables different behavior between 2.6.18 on fc6 and 2.6.23 on fc8

ebtables and iptables different behavior between 2.6.18 on fc6 and 2.6.23 on fc8

[Greg Scott]

This issue cost me most of a night's sleep and it will take me a few
paragraphs to describe.  The one sentence summary is, marking packets
with ebtables and reading later with iptables seems to behave
differently with 2.6.23 than it did with 2.6.18 and earlier.  

I put together a firewall system based on fc8 and the RedHat kernel that
ships with fc8.  I think this one is 2.6.23-2.  My previous latest and
greatest was with a 2.6.18 variant that shipped with fc6.  

This config requires that I bridge.  I need to bridge because I have
various H.323 videoconference codecs behind the firewall system and some
of these, especially older ones, do not work very well with NAT.  The
other choice besides bridging is proxy ARP, but I learned the hard way
that proxy ARP is horrible in colocation situations.  So I need to
bridge.  

But I also need NAT to handle traditional PCs and other stuff behind the
firewall.  

To make this work, I need to know which way packets are headed.  I need
to know on which interface packets come in, and which interface they go
out.  This is easy with routing, more challenging with bridging.
iptables had a facility that would let me test --physdev with bridged
packets.  It might still be there but I remember reading a discussion
about how this violated layering and would not be supported long term.  

So I found another way.  Ebtables lets me mark packets based on the
physical interface in or out.  In the broute table of ebtables, I can
mark packets in from the Internet with "1", and packets in from the LAN
side with "2".  I also need to know the interface on which packets go
out, so I mark those with "3" in the FORWARD and OUTPUT chains of the
filter table.  

And then later on, I can test the marks with iptables and take
appropriate actions depending on the direction of packet travel.  

Here is the ebtables code to do that, extracted from a very long
rc.firewall script - hopefully this won't be garbled in this post.  


echo "Flushing and zeroing all ebtables tables and chains"
$EBTABLES -t broute -F
$EBTABLES -t broute -Z
$EBTABLES -t filter -F
$EBTABLES -t filter -Z
$EBTABLES -t nat -F
$EBTABLES -t nat -Z

#
# Use ebtables to mark packets based on the in/out interface.
# 1 - (bit 0 set) for packets entering on the Internet physical
interface
# 2 - (bit 1 set) for packets entering on the trusted physical interface
# 3 - (bits 0 and 1) for packets exiting via the Internet physical
interface

echo "Marking bridged packets at layer 2 for later layer 3 filtering."
$EBTABLES -t broute -A BROUTING -i $INET_IFACE \
	-j mark --mark-set 1 --mark-target CONTINUE
$EBTABLES -t broute -A BROUTING -i $TRUSTED1_IFACE \
	-j mark --mark-set 2 --mark-target CONTINUE
$EBTABLES -t filter -A FORWARD -o $INET_IFACE \
	-j mark --mark-set 3 --mark-target CONTINUE
$EBTABLES -t filter -A OUTPUT -o $INET_IFACE \
	-j mark --mark-set 3 --mark-target CONTINUE


With that background, here is the problem.

None of my packets that were supposed to have that "3" mark ever kept
them.  For some reason, they either never were marked or they were
marked and then the mark disappeared.  The packets with "1" and "2"
worked just fine.  I set up tons of logging to chase that down.  But the
packets with "3" were not marked.  

Needless to say, this created major havoc with my network routing.  The
good news is, it only kept me up all night and none of the users at this
site ever knew anything was messed up.  

I built two nearly identical systems.  One with fc6, the other with fc8.
I set them up with the exact same IP Addresses and identical copies of
my rc.firewall script.  The only difference between them were various
logging and debug lines I inserted into my rc.firewall script at
different times to chase down the marking.  The fc6 system worked as I
expected, the fc8 system did not.

And now I can reproduce the problem here under controlled conditions.  

So my questions - what changed with the interaction betweem iptables and
ebtables?  Is there a supported way I can track the physical interfaces
on which packets come in and go out that I can count on for the
indefinite future?  Or did I uncover an obscure bug with
ebtables/iptables?  

I have test systems and a simulated environment right here so I can try
different kernel combinations if needed.  

Thanks

- Greg Scott

Re: ebtables and iptables different behavior between 2.6.18 on fc6 and 2.6.23 on fc8

[Jan Engelhardt]

On Tuesday 2008-03-25 02:18, Greg Scott wrote:

> This issue cost me most of a night's sleep and it will take me a few
> paragraphs to describe.  The one sentence summary is, marking packets
> with ebtables and reading later with iptables seems to behave
> differently with 2.6.23 than it did with 2.6.18 and earlier.
>
> I put together a firewall system based on fc8 and the RedHat kernel that
> ships with fc8.  I think this one is 2.6.23-2.  My previous latest and
> greatest was with a 2.6.18 variant that shipped with fc6.
>
> This config requires that I bridge.  I need to bridge because I have
> various H.323 videoconference codecs behind the firewall system and some
> of these, especially older ones, do not work very well with NAT.  The
> other choice besides bridging is proxy ARP, but I learned the hard way
> that proxy ARP is horrible in colocation situations.  So I need to
> bridge.
>
> But I also need NAT to handle traditional PCs and other stuff behind the
> firewall.
>
> To make this work, I need to know which way packets are headed.  I need
> to know on which interface packets come in, and which interface they go
> out.  This is easy with routing, more challenging with bridging.
> iptables had a facility that would let me test --physdev with bridged
> packets.  It might still be there but I remember reading a discussion
> about how this violated layering and would not be supported long term.

It still is, and I do not think it will change.
And actually, it is pretty easy, in two rules, unless I made a mistake 
somewhere:

 	# obviously bridged
 	iptables -t mangle -A FORWARD -i br0 -o br0 -j MARK --set-mark 111
 	(Or equivalently: ebtables -t filter -A FORWARD -p ipv4 -j mark
 	--set-mark 111)

 	# snat non-bridged
 	iptables -t nat -A POSTROUTING -o br0 -m physdev --physdev-out eth0
 		-m mark ! --mark 111 -j SNAT ...


> None of my packets that were supposed to have that "3" mark ever kept
> them.  For some reason, they either never were marked or they were
> marked and then the mark disappeared.

I can reproduce it. Testcase is actually a two-liner:

 	ebtables -A OUTPUT -o eth0 -j mark --set-mark 99
 	iptables -t mangle -A POSTROUTING -o br0 -j LOGMARK

LOGMARK (yay - this target really seems to show off :^)  tells me:

 	[214558.490202] nfmark=0x0 secmark=0x0 classify=0x2
 	ct=0xede74954 ctmark=0x0 ctstate=ESTABLISHED
 	ctstatus=,SEEN_REPLY,ASSURED,CONFIRMED

Now that is indeed interesting. I had this thought, maybe the mark does 
not disappear, maybe ebtables is run -- contrary to most graphics 
depicting the netfilter flow -- _after_ iptables. So I tried:

 	iptables -t mangle -A POSTROUTING -j LOG --log-prefix "[ipt] " -d 134.76.13.21
 	ebtables -A OUTPUT -p ipv4 --ip-destination 134.76.13.21 --log --log-prefix "[ebt] "

with the result of:

 	[214961.190130] [ipt] IN= OUT=br0 SRC=10.10.106.161 DST=134.76.13.21
 	LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0
 	ID=59252 SEQ=1

 	[214961.190186] [ebt]  IN= OUT=sis0 MAC source = 00:0a:e6:98:ed:d7 MAC
 	dest = 68:a8:3e:d3:d0:fb proto = 0x0800

which means ebtables actually comes after iptables, and hence, your mark 
3 will not show up as you expected.


> So my questions - what changed with the interaction betweem iptables and
> ebtables?

I'm going to figure out now...

Ebtables hook order anomaly (was: ebtables and iptables different behavior between 2.6.18 on fc6 and 2.6.23 on fc8)

[Jan Engelhardt]

On Tuesday 2008-03-25 03:28, Jan Engelhardt wrote:
>
>>  None of my packets that were supposed to have that "3" mark ever kept
>>  them.  For some reason, they either never were marked or they were
>>  marked and then the mark disappeared.
>
> I can reproduce it. [...]
> Now that is indeed interesting. I had this thought, maybe the mark does not 
> disappear, maybe ebtables is run -- contrary to most graphics depicting the 
> netfilter flow -- _after_ iptables. So I tried:
>
> 	 iptables -t mangle -A POSTROUTING -j LOG --log-prefix "[ipt] " -d
> 	 134.76.13.21
> 	 ebtables -A OUTPUT -p ipv4 --ip-destination 134.76.13.21 --log
> 	 --log-prefix "[ebt] "
>
> with the result of:
>
> 	 [214961.190130] [ipt] IN= OUT=br0 SRC=10.10.106.161 DST=134.76.13.21
> 	 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0
> 	 ID=59252 SEQ=1
>
> 	 [214961.190186] [ebt]  IN= OUT=sis0 MAC source = 00:0a:e6:98:ed:d7
> 	 MAC
> 	 dest = 68:a8:3e:d3:d0:fb proto = 0x0800
>
> which means ebtables actually comes after iptables, and hence, your mark 3 
> will not show up as you expected.
>
>>  So my questions - what changed with the interaction betweem iptables and
>>  ebtables?
>
> I'm going to figure out now...

I tested a 2.6.18 and experienced the same order of [ipt] and [ebt].
This means that trying to

 	$EBTABLES -t filter -A OUTPUT -o $INET_IFACE \
 	-j mark --mark-set 3 --mark-target CONTINUE

you will never see mark 3 (at least not when generated in ebt/OUTPUT)
in iptables again.


->

Altogether, this made me re-evaluate the Netfilter hook/table
ordering as to wtf is going on. The end result? I was surprised to
notice an odd order of packet flow inside Ebtables, depicted in the
updated image at http://jengelh.hopto.org/images/nf-packet-flow.png
it certainly explains things going not as expected on Greg's side.

I used -j TRACE for iptables and --log in ebtables to find 
the actual order, and something additionally has catched my 
eye.

This is the syslog dump on a router confused with an IP-less
br0 bridge enslaving rtl0 and vmnet1 (i.e. a pure passthru bridge).

From what can be observed, the ICMP Echo Reply packet (the last line
with 06:54:30, and also :31, and so on) is not hitting
TRACE:nat:POSTROUTING --- only the ICMP Echo Request is. How can this
happen?


[e-n-PR] = [ebtables-{nat|filter}-{PRerouting,Forward,POstrouting}]

06:54:30 [227559.984918] [e-n-PR]  IN=rtl0 OUT= MAC source = 00:0c:29:77:f2:bd MAC dest = 00:0c:29:a4:a5:b2 proto = 0x0800
06:54:30 [227559.984976] TRACE: raw:PREROUTING:policy:2 IN=br0 OUT= PHYSIN=rtl0 MAC=00:0c:29:a4:a5:b2:00:0c:29:77:f2:bd:08:00 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TY
06:54:30 [227559.985034] TRACE: mangle:PREROUTING:policy:1 IN=br0 OUT= PHYSIN=rtl0 MAC=00:0c:29:a4:a5:b2:00:0c:29:77:f2:bd:08:00 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP
06:54:30 [227559.985072] TRACE: nat:PREROUTING:policy:1 IN=br0 OUT= PHYSIN=rtl0 MAC=00:0c:29:a4:a5:b2:00:0c:29:77:f2:bd:08:00 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TY
06:54:30 [227559.985117] [e-f-F]  IN=rtl0 OUT=vmnet1 MAC source = 00:0c:29:77:f2:bd MAC dest = 00:0c:29:a4:a5:b2 proto = 0x0800
06:54:30 [227559.985141] TRACE: mangle:FORWARD:policy:3 IN=br0 OUT=br0 PHYSIN=rtl0 PHYSOUT=vmnet1 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=44042 SEQ=7 
06:54:30 [227559.985166] TRACE: filter:FORWARD:policy:5 IN=br0 OUT=br0 PHYSIN=rtl0 PHYSOUT=vmnet1 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=44042 SEQ=7 
06:54:30 [227559.985188] [e-n-PO]  IN= OUT=vmnet1 MAC source = 00:0c:29:77:f2:bd MAC dest = 00:0c:29:a4:a5:b2 proto = 0x0800
06:54:30 [227559.985211] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=br0 PHYSIN=rtl0 PHYSOUT=vmnet1 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=44042 SEQ=7 
06:54:30 [227559.985252] TRACE: nat:POSTROUTING:policy:3 IN= OUT=br0 PHYSIN=rtl0 PHYSOUT=vmnet1 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=44042 SEQ=7 
06:54:30 [227559.987300] [e-n-PR]  IN=vmnet1 OUT= MAC source = 00:0c:29:a4:a5:b2 MAC dest = 00:0c:29:77:f2:bd proto = 0x0800
06:54:30 [227559.987359] TRACE: raw:PREROUTING:policy:2 IN=br0 OUT= PHYSIN=vmnet1 MAC=00:0c:29:77:f2:bd:00:0c:29:a4:a5:b2:08:00 SRC=192.168.45.1 DST=192.168.45.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56525 PROTO=ICMP
06:54:30 [227559.987413] TRACE: mangle:PREROUTING:policy:1 IN=br0 OUT= PHYSIN=vmnet1 MAC=00:0c:29:77:f2:bd:00:0c:29:a4:a5:b2:08:00 SRC=192.168.45.1 DST=192.168.45.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56525 PROTO=I
06:54:30 [227559.987455] [e-f-F]  IN=vmnet1 OUT=rtl0 MAC source = 00:0c:29:a4:a5:b2 MAC dest = 00:0c:29:77:f2:bd proto = 0x0800
06:54:30 [227559.987479] TRACE: mangle:FORWARD:policy:3 IN=br0 OUT=br0 PHYSIN=vmnet1 PHYSOUT=rtl0 SRC=192.168.45.1 DST=192.168.45.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56525 PROTO=ICMP TYPE=0 CODE=0 ID=44042 SEQ=7 
06:54:30 [227559.987504] TRACE: filter:FORWARD:policy:5 IN=br0 OUT=br0 PHYSIN=vmnet1 PHYSOUT=rtl0 SRC=192.168.45.1 DST=192.168.45.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56525 PROTO=ICMP TYPE=0 CODE=0 ID=44042 SEQ=7 
06:54:30 [227559.987525] [e-n-PO]  IN= OUT=rtl0 MAC source = 00:0c:29:a4:a5:b2 MAC dest = 00:0c:29:77:f2:bd proto = 0x0800
06:54:30 [227559.987548] TRACE: mangle:POSTROUTING:policy:1 IN= OUT=br0 PHYSIN=vmnet1 PHYSOUT=rtl0 SRC=192.168.45.1 DST=192.168.45.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56525 PROTO=ICMP TYPE=0 CODE=0 ID=44042 SEQ=7
...repeats...
06:54:31 [227560.994489] [e-n-PR]  IN=rtl0 OUT= MAC source = 00:0c:29:77:f2:bd MAC dest = 00:0c:29:a4:a5:b2 proto = 0x0800
06:54:31 [227560.994548] TRACE: raw:PREROUTING:policy:2 IN=br0 OUT= PHYSIN=rtl0 MAC=00:0c:29:a4:a5:b2:00:0c:29:77:f2:bd:08:00 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TY
06:54:31 [227560.994609] TRACE: mangle:PREROUTING:policy:1 IN=br0 OUT= PHYSIN=rtl0 MAC=00:0c:29:a4:a5:b2:00:0c:29:77:f2:bd:08:00 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP
06:54:31 [227560.994646] TRACE: nat:PREROUTING:policy:1 IN=br0 OUT= PHYSIN=rtl0 MAC=00:0c:29:a4:a5:b2:00:0c:29:77:f2:bd:08:00 SRC=192.168.45.2 DST=192.168.45.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TY
06:54:31 [227560.994691] [e-f-F]  IN=rtl0 OUT=vmnet1 MAC source = 00:0c:29:77:f2:bd MAC dest = 00:0c:29:a4:a5:b2 proto = 0x0800
...
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> 
> On Tuesday 2008-03-25 03:28, Jan Engelhardt wrote:
>>
>>>  None of my packets that were supposed to have that "3" mark ever kept
>>>  them.  For some reason, they either never were marked or they were
>>>  marked and then the mark disappeared.
>>
>> I can reproduce it. [...]
>> Now that is indeed interesting. I had this thought, maybe the mark 
>> does not disappear, maybe ebtables is run -- contrary to most graphics 
>> depicting the netfilter flow -- _after_ iptables. So I tried:
>>
>>      iptables -t mangle -A POSTROUTING -j LOG --log-prefix "[ipt] " -d
>>      134.76.13.21
>>      ebtables -A OUTPUT -p ipv4 --ip-destination 134.76.13.21 --log
>>      --log-prefix "[ebt] "
>>
>> with the result of:
>>
>>      [214961.190130] [ipt] IN= OUT=br0 SRC=10.10.106.161 DST=134.76.13.21
>>      LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=ICMP TYPE=8 CODE=0
>>      ID=59252 SEQ=1
>>
>>      [214961.190186] [ebt]  IN= OUT=sis0 MAC source = 00:0a:e6:98:ed:d7
>>      MAC
>>      dest = 68:a8:3e:d3:d0:fb proto = 0x0800
>>
>> which means ebtables actually comes after iptables, and hence, your 
>> mark 3 will not show up as you expected.


Indeed, on output bridge netfilter will run after IPv4 netfilter.
Does that explain things?

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Tuesday 2008-03-25 13:57, Patrick McHardy wrote:
>> >
>> > which means ebtables actually comes after iptables, and hence,
>> > your mark 3 will not show up as you expected.
>
> Indeed, on output bridge netfilter will run after IPv4 netfilter.
> Does that explain things?

It explains things, but having the feature removed is not nice.
I cannot deny that the previous code was real a hook spaghetteria,
but how could it be done better?

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> On Tuesday 2008-03-25 13:57, Patrick McHardy wrote:
>>>> which means ebtables actually comes after iptables, and hence,
>>>> your mark 3 will not show up as you expected.
>> Indeed, on output bridge netfilter will run after IPv4 netfilter.
>> Does that explain things?
> 
> It explains things, but having the feature removed is not nice.
> I cannot deny that the previous code was real a hook spaghetteria,
> but how could it be done better?

Good question. The order on output is logically correct, so I
wouldn't change it. I never liked the invocation of IP hooks
from bridging at all, so long-term we could consider making
the features people want to bridging available "natively"
(like conntrack/NAT/...).

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Wednesday 2008-04-09 16:36, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> On Tuesday 2008-03-25 13:57, Patrick McHardy wrote:
>> > > > which means ebtables actually comes after iptables, and hence,
>> > > > your mark 3 will not show up as you expected.
>> > Indeed, on output bridge netfilter will run after IPv4 netfilter.
>> > Does that explain things?
>> 
>> It explains things, but having the feature removed is not nice.
>> I cannot deny that the previous code was real a hook spaghetteria,
>> but how could it be done better?
>
> Good question. The order on output is logically correct, so I
> wouldn't change it. I never liked the invocation of IP hooks
> from bridging at all, so long-term we could consider making
> the features people want to bridging available "natively"
> (like conntrack/NAT/...).

Can't quite imagine what you mean. Bridging is currently
implemented using a separate netdevice, much like bonding
or gre/ipip/tun/tap tunnels.

Well bridging is essentially a few pieces:

 - "forward" packets without changing MAC

 - arpreply engine

I have often come across needing a bridge device with a single port and
brouting=drop only because of ebt_arpreply; I have immediate plans
to remove this limitation.

 - STP

I seem to remember ideas about a userspace daemon were floating
around..

 - filtering on L2

there has been a L2-hooks back in the 2.4 days (at least the context
looks like it), and given that the ebtables targets now use
xtables, the l2hooks patch would actually be real easy.

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> On Wednesday 2008-04-09 16:36, Patrick McHardy wrote:
>> Jan Engelhardt wrote:
>>> It explains things, but having the feature removed is not nice.
>>> I cannot deny that the previous code was real a hook spaghetteria,
>>> but how could it be done better?
>> Good question. The order on output is logically correct, so I
>> wouldn't change it. I never liked the invocation of IP hooks
>> from bridging at all, so long-term we could consider making
>> the features people want to bridging available "natively"
>> (like conntrack/NAT/...).
> 
> Can't quite imagine what you mean. Bridging is currently
> implemented using a separate netdevice, much like bonding
> or gre/ipip/tun/tap tunnels.
> 
> Well bridging is essentially a few pieces:
> 
>  - "forward" packets without changing MAC
> 
>  - arpreply engine

A pure bridge doesn't need ARP. I don't get your point,
but my suggestion might not make much sense, I haven't
really thought about this.

> I have often come across needing a bridge device with a single port and
> brouting=drop only because of ebt_arpreply; I have immediate plans
> to remove this limitation.
> 
>  - STP
> 
> I seem to remember ideas about a userspace daemon were floating
> around..
> 
>  - filtering on L2
> 
> there has been a L2-hooks back in the 2.4 days (at least the context
> looks like it), and given that the ebtables targets now use
> xtables, the l2hooks patch would actually be real easy.

They were used for the kernel ct_sync implementation. Why
would you use them for briding, it has its own hooks?

RE: Ebtables hook order anomaly

[Greg Scott]

> A pure bridge doesn't need ARP. I don't get your point,
> but my suggestion might not make much sense, I haven't
> really thought about this.

My little bridge-router systems could use ARP.  I am putting together a
site right now that uses an active/passive HA pair of firewall
bridge/router systems.  I kludge in a MAC Address for the active system
to make the router in front of it happy, as well as sending out a GARP.


This site needs bridging because it has H.323 videoconferencing
equipment in place.  H.323 devices - especially older ones - do not get
along well with NAT.  So I give the H.323 stuff a public IP Address and
set up a bridge.  

This all worked nicely until the night before Easter when I tried a new
version and everything broke.  But I can live with not knowing the out
interface from L3 as long as I know the in-interface.  

- Greg

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Wednesday 2008-04-09 17:04, Patrick McHardy wrote:
>> 
>>  - arpreply engine
>
> A pure bridge doesn't need ARP. I don't get your point,
> but my suggestion might not make much sense, I haven't
> really thought about this.

Well maybe you don't want to use a bridge for simplicity
but still do proxy arp for lots of addresses.

>>  - filtering on L2
>> 
>> there has been a L2-hooks back in the 2.4 days (at least the context
>> looks like it), and given that the ebtables targets now use
>> xtables, the l2hooks patch would actually be real easy.
>
> They were used for the kernel ct_sync implementation. Why
> would you use them for briding, it has its own hooks?
>
If you wanted to filter out non-(arp,ipv4,ipv6) traffic,
you have to set up a bridge and filter it there, as I see it.
So L2 hooks would come in handy since you would not need
to set up a bridge device anymore to do so.

RE: Ebtables hook order anomaly

[Greg Scott]

> Well maybe you don't want to use a bridge for simplicity 
> but still do proxy arp for lots of addresses.

See the email I sent a few minutes ago about proxy ARP.  Proxy ARP
scares the daylights out of me.  Now, if there were a way to proxy ARP
for only a select set of IP Addresses - instead of ***all*** IP
Addresses hitting the interface - now that would be something!

- Greg

RE: Ebtables hook order anomaly

[Greg Scott]

> there has been a L2-hooks back in the 2.4 days (at least the context
> looks like it), and given that the ebtables targets now use
> xtables, the l2hooks patch would actually be real easy.

All I ask is, please please please don't break the order with ebtables
and iptables on input!  I really really really need to know the in-eth
interface.  

Also, I'm willing to test any l2-hook patches that tell me the iptables
out eth interface in a bridge.  I have a couple of sites where I can
brew up kernels and rulesets.

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> there has been a L2-hooks back in the 2.4 days (at least the context
>> looks like it), and given that the ebtables targets now use
>> xtables, the l2hooks patch would actually be real easy.
> 
> All I ask is, please please please don't break the order with ebtables
> and iptables on input!  I really really really need to know the in-eth
> interface.  

Don't worry.

> Also, I'm willing to test any l2-hook patches that tell me the iptables
> out eth interface in a bridge.  I have a couple of sites where I can
> brew up kernels and rulesets.

That won't work. iptables *can't* know since it sits at the network
layer, while briding sits *below* it at the device layer.

I forgot why exactly you need the bridge port in iptables. But
in any case the only way to get it is within briding (where some
of the iptables features are not available). So a fix for all
of this should be to make the missing iptables features available
for briding natively.

RE: Ebtables hook order anomaly

[Greg Scott]

> I forgot why exactly you need the bridge port in iptables.

We crossed in the email a minute ago when I described my H.323 stuff.  

There was another case a year or so ago that first turned me on to
bridging.  I inherited a network that had systems with IP Addresses all
over the place both in front and behind the firewall.  It was mess.  

I tried to set up a Linux firewall with proxy ARP but the results were
very ugly.  It turned out, they had this load balancer in their network
nobody told me about and my proxy ARP and the load-balancer's proxy ARPs
got confused with each other and this took down a popular website for an
hour or so at 5 in the morning central time.  

And then I finally figured out what proxy ARP really meant when my
outside eth0 interface was proxy-ARPing.  This was in a co-lo site and
there were other customers on the same Ethernet and I ended up
proxy-ARPing for them - woops!  This didn't cause any damage but it sure
scared me to death!

Needless to say, that was the last time I ever used proxy-ARP.  

So now I use bridging whenever I have devices that need public IP
Addresses inside a mostly NATed network.

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> I forgot why exactly you need the bridge port in iptables.
> 
> We crossed in the email a minute ago when I described my H.323 stuff.  

Yes, but I was wondering more about the exact rules where
this would be helpful. The things you described in your
other mail sounds easily doable by simply exluding port
1723 from NAT without any knowledge of bridge ports.

RE: Ebtables hook order anomaly

[Greg Scott]

>>> I forgot why exactly you need the bridge port in iptables.
>> 
>> We crossed in the email a minute ago when I described my H.323 stuff.


> Yes, but I was wondering more about the exact rules where
> this would be helpful. The things you described in your
> other mail sounds easily doable by simply exluding port
> 1723 from NAT without any knowledge of bridge ports.

I also need to know the direction - is the packet going back inside or
out to the Internet.  Same for input - is it coming from the inside or
coming in from the Internet.  Input is more important than output - but
knowing the output direction would be nice.  

Examples - 

I have a section in iptables-NAT-PREROUTING that tests on input for
anything from 10.n/8 or 172.16/12 or 192.168/16.  If in from the
Internet than I log and drop it.  Without knowing the in-interface, I
don't know how I would do those tests.  

Here is a forward rule for outbound stuff that broke with the recent
change:

$IPTABLES -A FORWARD -m mark --mark 3 -j ACCEPT
# Packets marked with 3 (bits 0 and 1 set) are outbound to the Internet.

The mark was for anything from the inside going out to the Internet.  

The FORWARD rules are not always that simple.  I have some sites that
need to regulate what can go out.  With the new ebtables order, I can
deal with that with IP Addresses instead of eth interfaces.  But it
would still be nice to have the ability to regulate by interface instead
of IP Address.  

Here was another one that made me scratch my bald head most of the night
before Easter - when this rule broke, you can imagine the weird
behavior:

echo "	MASQUERADing everything else going outbound"
##$IPTABLES -t nat -A POSTROUTING -m mark --mark 3 -j LOG --log-prefix
"mark3 "
$IPTABLES -t nat -A POSTROUTING -m mark --mark 3 -j MASQUERADE

I can change this to test for any IP Address other than the local LAN
and it should work.  It just makes things a little more complicated for
sites that have many local LANs.

- Greg

RE: Ebtables hook order anomaly

[Greg Scott]

> Indeed, on output bridge netfilter will run after IPv4 netfilter.
> Does that explain things?

Well, not really.  2.6.23-2 behaves differently than 2.6.18.  My
ebtables marks get set and read by iptables in 2.6.18.  iptables does
not see them in 2.6.23.

It could be something in the order of execution changed.  I'm using
RedHat kernels right now and I know they tweak the kernels a little bit.
But surely the RedHat guys would not change something this fundamental?


I set my ebtables mark for outbound packets in two places - in the
OUTPUT chain and the FORWARD chain.  I was looking mostly at the
ebtables FORWARD chain for packets that pass thru the box.  I set the
mark in ebtables FORWARD and OUTPUT.  Iptables POSTROUTING needs to see
them so it knows to MASQUERADE.  Otherwise, things get really weird.  :)
iptables sees the marks in 2.6.18, but does not see them in 2.6.23.  

I have a couple of 2.6.23 systems here and I can brew up a 2.6.18 fc6
system later on for testing if needed.  (All the 2.6.18 stuff I have
built up right now is in production.)  I can set up some tests similarly
to Jan's testing from last night for packets passing through the box.  

Or if there's a better way to do this, I'm open.  

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> Indeed, on output bridge netfilter will run after IPv4 netfilter.
>> Does that explain things?
> 
> Well, not really.  2.6.23-2 behaves differently than 2.6.18.  My
> ebtables marks get set and read by iptables in 2.6.18.  iptables does
> not see them in 2.6.23.
> 
> It could be something in the order of execution changed.  I'm using
> RedHat kernels right now and I know they tweak the kernels a little bit.
> But surely the RedHat guys would not change something this fundamental?


No, that was us :) Bridge-netfilter used to defer the IPv4 OUTPUT
and POSTROUTING hook until the outgoing bridge port was determined
by the bridge code. This "feature" was removed because it broke
all kinds of things, now the order matches the layering and IPv4
hooks are always processed entirely before bridging.

> I set my ebtables mark for outbound packets in two places - in the
> OUTPUT chain and the FORWARD chain.  I was looking mostly at the
> ebtables FORWARD chain for packets that pass thru the box.  I set the
> mark in ebtables FORWARD and OUTPUT.  Iptables POSTROUTING needs to see
> them so it knows to MASQUERADE.  Otherwise, things get really weird.  :)
> iptables sees the marks in 2.6.18, but does not see them in 2.6.23.  
> 
> I have a couple of 2.6.23 systems here and I can brew up a 2.6.18 fc6
> system later on for testing if needed.  (All the 2.6.18 stuff I have
> built up right now is in production.)  I can set up some tests similarly
> to Jan's testing from last night for packets passing through the box.  
> 
> Or if there's a better way to do this, I'm open.  

For routed packets you can't make any decisions in iptables based
on the outgoing bridge port, thats only possible for purely bridged
traffic.

Re: Ebtables hook order anomaly

[Jan Engelhardt]

On Tuesday 2008-03-25 14:49, Patrick McHardy wrote:
> Greg Scott wrote:
>>
>>  It could be something in the order of execution changed.  I'm using
>>  RedHat kernels right now and I know they tweak the kernels a little bit.
>>  But surely the RedHat guys would not change something this fundamental?
>
> No, that was us :) Bridge-netfilter used to defer the IPv4 OUTPUT

Do you have the commit id at hand? Was it
2bf540b73ed5b304e84bb4d4c390d49d1cfa0ef8?

> and POSTROUTING hook until the outgoing bridge port was determined
> by the bridge code. This "feature" was removed because it broke
> all kinds of things, now the order matches the layering and IPv4
> hooks are always processed entirely before bridging.

Now the order is .. non-consistent.
On a pure bridge forward (-i br -o br), as I have determined,
ebtables-nat-POSTROUTING comes _before_ the IPv4 hooks.

Re: Ebtables hook order anomaly

[Patrick McHardy]

Jan Engelhardt wrote:
> 
> On Tuesday 2008-03-25 14:49, Patrick McHardy wrote:
>> Greg Scott wrote:
>>>
>>>  It could be something in the order of execution changed.  I'm using
>>>  RedHat kernels right now and I know they tweak the kernels a little 
>>> bit.
>>>  But surely the RedHat guys would not change something this fundamental?
>>
>> No, that was us :) Bridge-netfilter used to defer the IPv4 OUTPUT
> 
> Do you have the commit id at hand? Was it
> 2bf540b73ed5b304e84bb4d4c390d49d1cfa0ef8?

Yes.

>> and POSTROUTING hook until the outgoing bridge port was determined
>> by the bridge code. This "feature" was removed because it broke
>> all kinds of things, now the order matches the layering and IPv4
>> hooks are always processed entirely before bridging.
> 
> Now the order is .. non-consistent.
> On a pure bridge forward (-i br -o br), as I have determined,
> ebtables-nat-POSTROUTING comes _before_ the IPv4 hooks.

Thats indeed inconsistent. I don't believe this has changed
however, the IPv4 POSTROUTING hook was always called from
the bridge POSTROUTING hook (with similar priorities).

RE: Ebtables hook order anomaly

[Greg Scott]

> For routed packets you can't make any decisions in iptables based on 
> the outgoing bridge port, thats only possible for purely bridged 
> traffic.

Bummer - but at least I now know the rules.  I guess I'll test using
source IP Address.  That kind of works - I know what stuff is on either
side - but it's messy.  

Thanks

- GReg

RE: Ebtables hook order anomaly

[Greg Scott]

> Thats indeed inconsistent. I don't believe this has changed however, 
> the IPv4 POSTROUTING hook was always called from the bridge 
> POSTROUTING hook (with similar priorities).

Well, OK - so I guess the most important question now is, are there any
other behavior changes of this nature coming up so I can get ready for
them in advance?  I'm kind of hoping for no more rude midnight
surprises.  :)

Thanks

- Greg

Re: Ebtables hook order anomaly

[Patrick McHardy]

Greg Scott wrote:
>> Thats indeed inconsistent. I don't believe this has changed however, 
>> the IPv4 POSTROUTING hook was always called from the bridge 
>> POSTROUTING hook (with similar priorities).
> 
> Well, OK - so I guess the most important question now is, are there any
> other behavior changes of this nature coming up so I can get ready for
> them in advance?  I'm kind of hoping for no more rude midnight
> surprises.  :)

Looking at feature-removal-schedule, not from my side :)

RE: Ebtables hook order anomaly

[Greg Scott]

>>> and POSTROUTING hook until the outgoing bridge port was determined
by 
>>> the bridge code. This "feature" was removed because it broke all 
>>> kinds of things, now the order matches the layering and IPv4 hooks 
>>> are always processed entirely before bridging.
>> 
>> Now the order is .. non-consistent.
>> On a pure bridge forward (-i br -o br), as I have determined, 
>> ebtables-nat-POSTROUTING comes _before_ the IPv4 hooks.
>
> Thats indeed inconsistent. I don't believe this has changed however, 
> the IPv4 POSTROUTING hook was always called from the bridge
POSTROUTING 
> hook (with similar priorities).

This woke me up in the middle of the night - I also mark packets in
ebtables BROUTE based on the incoming interface and then test all over
the place in iptables based on that mark.  One of the most important is
a test for -s {private IP Address} coming in from the Internet.  But
there are lots of other tests based on source IP and incoming interface.
I really really really need to know the incoming interface.  

This still seems to work - ebtables BROUTE still seems to come before
iptables NAT PREROUTING and my ebtables BROUTE marks all show up in
iptables.  Am I on solid ground for the future if I keep this up?

Thanks

- Greg

RE: Ebtables hook order anomaly

[Greg Scott]

> This still seems to work - ebtables BROUTE still seems to come 
> before iptables NAT PREROUTING and my ebtables BROUTE marks all 
> show up in iptables.  Am I on solid ground for the future if I 
> keep this up?

So can I count on this working for the long term?

ie - Can I mark packets in ebtables-BROUTE based on the in-interface and
then read the mark later on in iptables?  (I already know I can't do
this any more for the out-interface.)

Thanks

- Greg