ebtables broute DROP problem in production environment

ebtables broute DROP problem in production environment

[Felipe W Damasio]

  Hi all,

  I'm trying to use ebtables/iptables to implement a tproxy-squid on my network.

  I have a bridge-setup, with eth0 facing the user and eth1 facing the internet.

  Everything works great when I have an user connected with a
cross-over cable on eth0.

  But when I plug eth0 on the production environment network (which
uses multiple VLANs, one for the users and another for the internet),
http traffic stop working (ie. doesn't get routed to squid).

  We use a Cisco switch 2690 (layer 2).

  I'm trying to figure out what's wrong with my setup:

uname -r : 2.6.29.6
ebtables --version : ebtables v2.0.9-1 (June 2009)
iptables --version : iptables v1.4.3.2

Rules applied:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING  -i eth0 -p tcp --dport 80  -j TPROXY
--tproxy-mark 0x1/0x1 --on-port 3128
ebtables -t broute -A BROUTING -i eth0 -p ipv4 --ip-proto tcp
--ip-dport 80 -j redirect --redirect-target DROP
ebtables -t broute -A BROUTING -i eth1 -p ipv4 --ip-proto tcp
--ip-sport 80 -j redirect --redirect-target DROP

 cd /proc/sys/net/bridge/
 for i in *
 do
   echo 0 > $i
 done
 unset i

brctl stp br0 off
brctl setfd br0 1
brctl sethello br0 1
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
echo 1 >  /proc/sys/net/ipv4/ip_forward
echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
echo 1 >  /proc/sys/net/ipv4/tcp_low_latency
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/br0/rp_filter

  What am I missing?

  FYI, I tried using 2.6.32.2 and didn't work the ebtables even with
cross-over dummy client. Same with 2.6.33-rc1 and -rc1-git3.  The
kernel 2.6.29.6 was the only one I tried that worked correctly.

  If you need any other info, please let me know.

  Thanks in advance!

Felipe Damasio

Re: ebtables broute DROP problem in production environment

[Pascal Hambourg]

Hello,

Felipe W Damasio a écrit :
> 
>   I'm trying to use ebtables/iptables to implement a tproxy-squid on my network.
> 
>   I have a bridge-setup, with eth0 facing the user and eth1 facing the internet.
> 
>   Everything works great when I have an user connected with a
> cross-over cable on eth0.
> 
>   But when I plug eth0 on the production environment network (which
> uses multiple VLANs, one for the users and another for the internet),
> http traffic stop working (ie. doesn't get routed to squid).

Do you mean that eth0 sees VLAN tagged traffic ?

Re: ebtables broute DROP problem in production environment

[Felipe W Damasio]

  Hi,

2009/12/23 Felipe W Damasio <felipewd@gmail.com>:
>   But when I plug eth0 on the production environment network (which
> uses multiple VLANs, one for the users and another for the internet),
> http traffic stop working (ie. doesn't get routed to squid).

  One other thing: I tried using --log-level debug --log-ip log--arp
on the ebtables rules, and had several entries on my syslog such as
this:

Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
= 00:21:a0:ce:9d:24 MAC dest = 00:1a:a2:5d:70:8d proto = 0x0800 IP
SRC=189.10.205.122 IP DST=189.73.192.220, IP tos=0x00, IP proto=6
SPT=3774 DPT=80
Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
= 00:21:a0:ce:9d:24 MAC dest = 00:1a:a2:5d:70:8d proto = 0x0800 IP
SRC=189.10.204.12 IP DST=64.233.163.86, IP tos=0x00, IP proto=6
SPT=1260 DPT=80
Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
= 00:21:a0:ce:9d:24 MAC dest = 00:1d:71:b0:23:11 proto = 0x0800 IP
SRC=189.58.246.156 IP DST=72.21.81.133, IP tos=0x00, IP proto=6
SPT=2253 DPT=80
Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
= 00:21:a0:ce:9d:24 MAC dest = 00:1d:71:b0:23:11 proto = 0x0800 IP
SRC=189.58.247.99 IP DST=69.175.26.18, IP tos=0x00, IP proto=6
SPT=49392 DPT=80
Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
= 00:21:a0:ce:9d:24 MAC dest = 00:1a:a2:5d:70:8d proto = 0x0800 IP
SRC=201.66.236.140 IP DST=174.140.128.6, IP tos=0x00, IP proto=6
SPT=2060 DPT=80

  I suppose it means that the ebtables rules are working. But why
aren't they seen by the iptables rules?

  Again, I tried using a single cross-cable connected machine and
these rules worked (and got logged just the the above).

  Could this be a kernel bug?

  Cheers,

Felipe Damasio

Re: ebtables broute DROP problem in production environment

[Felipe W Damasio]

  Hi All,

  Mr. Pascal, I'm sorry, I'm not subscribed to this list...so I just
saw your reply on the archives.

  The thing is, I narrowed down the problem:

  - The traffic is passing through the bridge just fine;
  - When I plug a single client everything works great;
  - When I plug in the CMTS (all the cable modem clients, then),
everything stops.

  So, first I thought that the CMTS must be doing something to the net
to upset ebtables.

  But I added a rule:

iptables -A INPUT -p tcp --dport 80 -j LOG --log-level 1 --log-prefix
"iptables "

  And I got _a lot_ of these:

Dec 29 20:05:16 hyper kernel: iptables IN=eth0 OUT=
MAC=00:ea:01:02:7b:a2:00:21:a0:ce:9d:24:08:00 SRC=200.250.249.216
DST=201.49.208.251 LEN=52 TOS=0x00 PREC=0x00 TTL=63 ID=40080 DF
PROTO=TCP SPT=2959 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0 MARK=0x1

  So it's not VLAN-related.

  So now I'm thinking: If squid isn't seeing anything, couldn't be
that when I plug all the clients (around 6000) some buffer overflows
(maybe a proc entry?) and ebtables/iptables stop routing?

  I still get the logs on /var/log/messages, but squid doesn't get anything.

  Is there some proc entries I should check out?

  So far, the only one I changed to get the bridge up and running size-wise was:

echo 1000000 > /proc/sys/net/ipv4/netfilter/ip_conntrack_max

  Anything else is pretty much vanilla-default.

  If you guys could please CC me on the reply, I'd appreciate it.

  Thanks!

Felipe Damasio

2009/12/24 Felipe W Damasio <felipewd@gmail.com>:
>  Hi,
>
> 2009/12/23 Felipe W Damasio <felipewd@gmail.com>:
>>   But when I plug eth0 on the production environment network (which
>> uses multiple VLANs, one for the users and another for the internet),
>> http traffic stop working (ie. doesn't get routed to squid).
>
>  One other thing: I tried using --log-level debug --log-ip log--arp
> on the ebtables rules, and had several entries on my syslog such as
> this:
>
> Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
> = 00:21:a0:ce:9d:24 MAC dest = 00:1a:a2:5d:70:8d proto = 0x0800 IP
> SRC=189.10.205.122 IP DST=189.73.192.220, IP tos=0x00, IP proto=6
> SPT=3774 DPT=80
> Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
> = 00:21:a0:ce:9d:24 MAC dest = 00:1a:a2:5d:70:8d proto = 0x0800 IP
> SRC=189.10.204.12 IP DST=64.233.163.86, IP tos=0x00, IP proto=6
> SPT=1260 DPT=80
> Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
> = 00:21:a0:ce:9d:24 MAC dest = 00:1d:71:b0:23:11 proto = 0x0800 IP
> SRC=189.58.246.156 IP DST=72.21.81.133, IP tos=0x00, IP proto=6
> SPT=2253 DPT=80
> Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
> = 00:21:a0:ce:9d:24 MAC dest = 00:1d:71:b0:23:11 proto = 0x0800 IP
> SRC=189.58.247.99 IP DST=69.175.26.18, IP tos=0x00, IP proto=6
> SPT=49392 DPT=80
> Dec 23 19:24:47 hyper kernel: ebtables-broute IN=eth0 OUT= MAC source
> = 00:21:a0:ce:9d:24 MAC dest = 00:1a:a2:5d:70:8d proto = 0x0800 IP
> SRC=201.66.236.140 IP DST=174.140.128.6, IP tos=0x00, IP proto=6
> SPT=2060 DPT=80
>
>  I suppose it means that the ebtables rules are working. But why
> aren't they seen by the iptables rules?
>
>  Again, I tried using a single cross-cable connected machine and
> these rules worked (and got logged just the the above).
>
>  Could this be a kernel bug?
>
>  Cheers,
>
> Felipe Damasio
>

Re: ebtables broute DROP problem in production environment

[Felipe W Damasio]

  Hi All,

2009/12/29 Felipe W Damasio <felipewd@gmail.com>:
>  - The traffic is passing through the bridge just fine;
>  - When I plug a single client everything works great;
>  - When I plug in the CMTS (all the cable modem clients, then),
> everything stops.

  I now see that iptables -t mangle -L -v shows growing matches:

pkts bytes target     prot opt in     out     source
destination
 7508  466K DIVERT     tcp  --  any    any     anywhere
anywhere            socket
37713   13M TPROXY     tcp  --  eth0   any     anywhere
anywhere            tcp dpt:http TPROXY redirect 0.0.0.0:3128 mark
0x1/0x1

  So the traffic _seems_ to be routed to port 3128. But how can I make
sure that squid is getting it?

  I don't get why it's not working....since it works with a single user.

  Thanks,

Felipe Damasio