Fw: shaping vlans - revisited

Fw: shaping vlans - revisited

[Ethy H. Brito]

Date: Thu, 3 Feb 2011 15:28:59 -0200
From: "Ethy H. Brito" <ethy.brito@inexo.com.br>
To: Marek Kierdelewicz <marek@piasta.pl>
Subject: Re: shaping vlans - revisited


On Thu, 3 Feb 2011 16:46:53 +0100
Marek Kierdelewicz <marek@piasta.pl> wrote:

> >Hi All
> 
> Hi,
> 
> >I can see the counters increase (tc -s ...) for classid 1:2345 (eth0)
> >but they are zero for classid 2:2345 (eth1 - which the vlans are
> >attached in)
> 
> Are you seeing any traffic on root class on eth1? 

Yes I am.

> What's the output of following commang:
> 
> tc -s -d class sh dev eth1|grep -A 5 'root'

as requested:

# tc -s -d class sh dev eth1|grep -A 5 'root'
class htb 2:1 root rate 102400Kbit ceil 102400Kbit burst 52992b/8 mpu 0b overhead 0b cburst 52992b/8 mpu 0b overhead 0b level 7 
 Sent 8736060613 bytes 26878684 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 3463Kbit 1547pps backlog 0b 0p requeues 0 
 lended: 0 borrowed: 0 giants: 0
 tokens: 4039 ctokens: 4039



-- 

Ethy H. Brito         /"\
InterNexo Ltda.       \ /  CAMPANHA DA FITA ASCII - CONTRA MAIL HTML
+55 (12) 3797-6860     X   ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
S.J.Campos - Brasil   / \ 
 
PGP key: http://www.inexo.com.br/~ethy/0xC3F222A0.asc

Re: Fw: shaping vlans - revisited

[Andrew Beverley]

On Thu, 2011-02-03 at 15:29 -0200, Ethy H. Brito wrote:
> On Thu, 3 Feb 2011 16:46:53 +0100
> Marek Kierdelewicz <marek@piasta.pl> wrote:
> 
> > >Hi All
> > 
> > Hi,
> > 
> > >I can see the counters increase (tc -s ...) for classid 1:2345 (eth0)
> > >but they are zero for classid 2:2345 (eth1 - which the vlans are
> > >attached in)
> > 
> > Are you seeing any traffic on root class on eth1? 
> 
> Yes I am.
> 

Are you sure you are marking packets correctly *both* ways? Just because
they are being marked in one direction, doesn't mean that they are being
marked in the other direction.

If you add an iptables rule to LOG your traffic on that interface, then
you will be able to see the mark value.

Otherwise, if you're seeing traffic in the root class but not in the
leaf class, then that would imply that there is something wrong with the
filter.

Andy

Re: shaping vlans - revisited

[Ethy H. Brito]

On Thu, 03 Feb 2011 18:50:48 +0000
Andrew Beverley <andy@andybev.com> wrote:

> On Thu, 2011-02-03 at 15:29 -0200, Ethy H. Brito wrote:
> > On Thu, 3 Feb 2011 16:46:53 +0100
> > Marek Kierdelewicz <marek@piasta.pl> wrote:
> > 
> > > >Hi All
> > > 
> > > Hi,
> > > 
> > > >I can see the counters increase (tc -s ...) for classid 1:2345
> > > >(eth0) but they are zero for classid 2:2345 (eth1 - which the vlans
> > > >are attached in)
> > > 
> > > Are you seeing any traffic on root class on eth1? 
> > 
> > Yes I am.
> > 
> 
> Are you sure you are marking packets correctly *both* ways? Just because
> they are being marked in one direction, doesn't mean that they are being
> marked in the other direction.

Andrew 

If you see my first post you will find the filter rule I use to insert the
packet in each flowid according to its IPMARK. 

The marks is performed by these four lines:

/usr/sbin/iptables -t mangle -A POSTROUTING -o eth0 -m mark ! --mark 0 \
	-j IPMARK --addr dst --and-mask 0xffff --or-mask 0x20000
/usr/sbin/iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0 \
	-j IPMARK --addr dst --and-mask 0xffff --or-mask 0x0
/usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark ! --mark 2 \
	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x0
/usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark --mark 2 \
	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x40000

and since it is working for eth0 I assume they are correct for both
interfaces.

The script does the same thing to both eth0 and eth1 interfaces.

One extra detail is: this setup was working (just as it is for eth0) until
two weeks ago when then we created the vlans and splited the traffic among
them.

> 
> If you add an iptables rule to LOG your traffic on that interface, then
> you will be able to see the mark value.

This is a problem! 

If I run: iptables -I FORWARD -o eth1 -j LOG
I see NO traffic at all!! Nothing!!

But: iptables -I FORWARD -o vlan+ -j LOG
gives me tons of logs!!

iptables only see the traffic through vlans and not through eth1.

The same for tcpdump!!

BTW "mark 2" is there in the iptable LOG.
IN=eth0 OUT=vlan3 SRC=...blablabla ... ACK PSH URGP=0 MARK=0x2

But where can I find IPMARK??

> Otherwise, if you're seeing traffic in the root class but not in the
> leaf class, then that would imply that there is something wrong with the
> filter.

I believe that. I just can not see what is wrong!!

Ethy

Re: shaping vlans - revisited

[Andrew Beverley]

> > Are you sure you are marking packets correctly *both* ways? Just because
> > they are being marked in one direction, doesn't mean that they are being
> > marked in the other direction.
> 
> Andrew 
> 
> If you see my first post you will find the filter rule I use to insert the
> packet in each flowid according to its IPMARK. 
> 
> The marks is performed by these four lines:
> 
> /usr/sbin/iptables -t mangle -A POSTROUTING -o eth0 -m mark ! --mark 0 \
> 	-j IPMARK --addr dst --and-mask 0xffff --or-mask 0x20000
> /usr/sbin/iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0 \
> 	-j IPMARK --addr dst --and-mask 0xffff --or-mask 0x0
> /usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark ! --mark 2 \
> 	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x0
> /usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark --mark 2 \
> 	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x40000
> 
> and since it is working for eth0 I assume they are correct for both
> interfaces.

Okay.

> 
> The script does the same thing to both eth0 and eth1 interfaces.
> 
> One extra detail is: this setup was working (just as it is for eth0) until
> two weeks ago when then we created the vlans and splited the traffic among
> them.
> 
> > 
> > If you add an iptables rule to LOG your traffic on that interface, then
> > you will be able to see the mark value.
> 
> This is a problem! 
> 
> If I run: iptables -I FORWARD -o eth1 -j LOG
> I see NO traffic at all!! Nothing!!
> 
> But: iptables -I FORWARD -o vlan+ -j LOG
> gives me tons of logs!!
> 
> iptables only see the traffic through vlans and not through eth1.
> 

I have to admit I've never played with vlans and iptables, so I was just
guessing based on non-vlan experience.

Stupid question - can you not just attach your qdisc to the vlan
interface and mark the traffic on the vlan interface as well, as if it
was eth1?

Alternatively, if the root qdisc is seeing all the packets but it's the
marking that's not working, then can you not just mark on the vlan
interface rather than eth1?

Andy

Re: shaping vlans - revisited

[Marek Kierdelewicz]

>then can you not just mark on the vlan interface rather than eth1?

Andrew is right. Thats the best option. Ethy, just change eth1 to vlan+
in following rules:

/usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark ! --mark 2 \
	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x0
/usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark --mark 2 \
	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x40000

Best regards,
Marek Kierdelewicz

Re: shaping vlans - revisited

[Ethy H. Brito]

On Fri, 04 Feb 2011 02:59:16 +0000
Andrew Beverley <andy@andybev.com> wrote:


> Stupid question - can you not just attach your qdisc to the vlan
> interface and mark the traffic on the vlan interface as well, as if it
> was eth1?  

You mean "tc qdisc add dev vlanX ..." + "iptables ... -j IPMARK ..." ??

No. This must not be done. I have now 6 vlans (3 for IPv4 and 3 for IPv6)
each pair to one ISP. The only way I know (if you guys know another way,
please advise) to shape *_global_* aggregated traffic for each client is to attach to eth1 an not to individual vlans. 

> 
> Alternatively, if the root qdisc is seeing all the packets but it's the
> marking that's not working, then can you not just mark on the vlan
> interface rather than eth1?  

# iptables -L POSTROUTING -t mangle -v
Chain POSTROUTING (policy ACCEPT 3868M packets, 1901G bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2257  404K IPMARK     all  --  any    eth0    anywhere             anywhere            mark match !0x0 IPMARK dst ip and 0xffff or 0x20000
58092   46M IPMARK     all  --  any    eth0    anywhere             anywhere            mark match 0x0 IPMARK dst ip and 0xffff or 0x0
  997  117K IPMARK     all  --  any    vlan+   anywhere             anywhere            mark match !0x2 IPMARK src ip and 0xffff or 0x0
43665 6668K IPMARK     all  --  any    vlan+   anywhere
anywhere            mark match 0x2 IPMARK src ip and 0xffff or 0x40000

But...

# tc -s filter sh dev eth1|grep success
  mark 0x2101 0xffff (success 0)
  mark 0x2e03 0xffff (success 0)
  mark 0x2288 0xffff (success 0)
  mark 0x2012 0xffff (success 0)
  mark 0x2645 0xffff (success 0)
  mark 0x2108 0xfff8 (success 0)
(Very long list follows)

I am pretty sure that or the marking or filtering  (or both!) is not
working.
I can see traffic through root class and through default htb class
(2:ffff).
All traffic is through default class telling me marking/filtering is not
ok.

class htb 2:1 root rate 102400Kbit ceil 102400Kbit burst 52992b cburst 52992b 
 Sent 92926111 bytes 500251 pkt (dropped 0, overlimits 0 requeues 0) 

class htb 2:ffff parent 2:1 leaf ffff: prio 7 rate 5145Kbit ceil 5145Kbit burst 4181b cburst 4181b 
 Sent 93253253 bytes 500253 pkt (dropped 1037, overlimits 0 requeues 0) 


Ethy

Re: shaping vlans - revisited

[Andrew Beverley]

On Fri, 2011-02-04 at 12:05 -0200, Ethy H. Brito wrote:
> On Fri, 04 Feb 2011 02:59:16 +0000
> Andrew Beverley <andy@andybev.com> wrote:
> 
> 
> > Stupid question - can you not just attach your qdisc to the vlan
> > interface and mark the traffic on the vlan interface as well, as if it
> > was eth1?  
> 
> You mean "tc qdisc add dev vlanX ..." + "iptables ... -j IPMARK ..." ??
> 
> No. This must not be done. I have now 6 vlans (3 for IPv4 and 3 for
>  IPv6) each pair to one ISP. The only way I know (if you guys know
>  another way, please advise) to shape *_global_* aggregated traffic for
>  each client is to attach to eth1 an not to individual vlans. 
> 

Could you use an IFB device? Or possibly even IMQ?

Andy

Re: shaping vlans - revisited

[Ethy H. Brito]

On Fri, 04 Feb 2011 14:24:05 +0000
Andrew Beverley <andy@andybev.com> wrote:

> On Fri, 2011-02-04 at 12:05 -0200, Ethy H. Brito wrote:
> > On Fri, 04 Feb 2011 02:59:16 +0000
> > Andrew Beverley <andy@andybev.com> wrote:
> > 
> > 
> > > Stupid question - can you not just attach your qdisc to the vlan
> > > interface and mark the traffic on the vlan interface as well, as if
> > > it was eth1?  
> > 
> > You mean "tc qdisc add dev vlanX ..." + "iptables ... -j IPMARK ..." ??
> > 
> > No. This must not be done. I have now 6 vlans (3 for IPv4 and 3 for
> >  IPv6) each pair to one ISP. The only way I know (if you guys know
> >  another way, please advise) to shape *_global_* aggregated traffic for
> >  each client is to attach to eth1 an not to individual vlans. 
> > 
> 
> Could you use an IFB device? Or possibly even IMQ?

IFB?? 

it may be a good way out.
Do you know any discussion list or forum for it? 
I just read a few sites and have some questions.

Ethy

Re: shaping vlans - revisited

[Andrew Beverley]

On Fri, 2011-02-04 at 14:24 -0200, Ethy H. Brito wrote:
> On Fri, 04 Feb 2011 14:24:05 +0000
> Andrew Beverley <andy@andybev.com> wrote:
> 
> > On Fri, 2011-02-04 at 12:05 -0200, Ethy H. Brito wrote:
> > > On Fri, 04 Feb 2011 02:59:16 +0000
> > > Andrew Beverley <andy@andybev.com> wrote:
> > > 
> > > 
> > > > Stupid question - can you not just attach your qdisc to the vlan
> > > > interface and mark the traffic on the vlan interface as well, as if
> > > > it was eth1?  
> > > 
> > > You mean "tc qdisc add dev vlanX ..." + "iptables ... -j IPMARK ..." ??
> > > 
> > > No. This must not be done. I have now 6 vlans (3 for IPv4 and 3 for
> > >  IPv6) each pair to one ISP. The only way I know (if you guys know
> > >  another way, please advise) to shape *_global_* aggregated traffic for
> > >  each client is to attach to eth1 an not to individual vlans. 
> > > 
> > 
> > Could you use an IFB device? Or possibly even IMQ?
> 
> IFB?? 
> 
> it may be a good way out.
> Do you know any discussion list or forum for it? 

This list is probably a good place to start. There doesn't seem to be a
huge amount of information around for it, but I got a good response from
the author when I contacted him with a question a while back.

Andy

Re: shaping vlans - revisited

[Marek Kierdelewicz]

Hi,

>IFB?? 
>
>it may be a good way out.
>Do you know any discussion list or forum for it? 
>I just read a few sites and have some questions.

You could do it this way:

1) make sure ifb is modprobed and interface ifb0 is up

2) redirect outgoing traffic on each vlan interface to ifb0

tc qdisc add dev $DEV root handle 1: htb default 1
tc class add dev $DEV parent 1: classid 1:1 htb rate 100mbit
rc filter add dev $DEV protocol ip parent 1: prio 4 u32 \
  match ip dst 0.0.0.0/0 flowid :1 \
  action mirred egress redirect dev ifb0

... where $DEV is vlanX, vlanY ...

3) Add shaping on ifb0 just like you did earlier on eth1.

Best regards,
Marek Kierdelewicz

Re: shaping vlans - revisited

[Ethy H. Brito]

On Sat, 5 Feb 2011 13:46:21 +0100
Marek Kierdelewicz <marek@piasta.pl> wrote:

Hi Marek

I thing we will only change the place of the problem. It will move from
eth to ifb.

As stated by Grant Taylor in another post, the problem is my filters are
not redirecting packet to the flowids because "protocol ip" leads to
false. This should be someting like "protocol ieee802.1q".

I could not find such protocol parameter (or something that replaces it)
and tc filter demands a "protocol foo" I think due to my u32 classifiers.

Regards

Ethy




> Hi,
> 
> >IFB?? 
> >
> >it may be a good way out.
> >Do you know any discussion list or forum for it? 
> >I just read a few sites and have some questions.
> 
> You could do it this way:
> 
> 1) make sure ifb is modprobed and interface ifb0 is up
> 
> 2) redirect outgoing traffic on each vlan interface to ifb0
> 
> tc qdisc add dev $DEV root handle 1: htb default 1
> tc class add dev $DEV parent 1: classid 1:1 htb rate 100mbit
> rc filter add dev $DEV protocol ip parent 1: prio 4 u32 \
>   match ip dst 0.0.0.0/0 flowid :1 \
>   action mirred egress redirect dev ifb0
> 
> ... where $DEV is vlanX, vlanY ...
> 
> 3) Add shaping on ifb0 just like you did earlier on eth1.
> 
> Best regards,
> Marek Kierdelewicz


-- 

Ethy H. Brito         /"\
InterNexo Ltda.       \ /  CAMPANHA DA FITA ASCII - CONTRA MAIL HTML
+55 (12) 3797-6860     X   ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL
S.J.Campos - Brasil   / \
 
PGP key: http://www.inexo.com.br/~ethy/0xC3F222A0.asc

Re: shaping vlans - revisited

[Marek Kierdelewicz]

>Hi Marek

Hi,

>I thing we will only change the place of the problem. It will move from
>eth to ifb.

It'll be ok, because you'll be redirecting traffic from vlan
interface not eth1 interface. TC will see traffic without extra
vlan tag.

I use such setup to aggregate and shape traffic from many ppp
interfaces (PPPoE sessions) and it works ok. TC u32 filters properly
classify ip packets. Im my case redirection occurs on ppp interfaces,
not vlan interfaces but that's analogous.

Best regards,
Marek Kierdelewicz

Re: shaping vlans - revisited

[Ethy H. Brito]

On Sat, 5 Feb 2011 15:52:02 +0100
Marek Kierdelewicz <marek@piasta.pl> wrote:

> >Hi Marek
> 
> Hi,
> 
> >I thing we will only change the place of the problem. It will move from
> >eth to ifb.
> 
> It'll be ok, because you'll be redirecting traffic from vlan
> interface not eth1 interface. TC will see traffic without extra
> vlan tag.

... and since the packets are not encapsulated yet, how they will return
from ifb to vlanX to be encapsulated??

I see no tc command doing this in you example in your previous post.
Is there a missing command?

Ethy

Re: shaping vlans - revisited

[Marek Kierdelewicz]

Hi,

>... and since the packets are not encapsulated yet, how they will
>return from ifb to vlanX to be encapsulated??
>I see no tc command doing this in you example in your previous post.
>Is there a missing command?

That's the way ifb works. Example of ingress shaping that uses "action
mirred egress redirect":
http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb#Typical_Usage

Best regards,
Marek Kierdelewicz

Re: shaping vlans - revisited

[Ethy H. Brito]

On Sat, 5 Feb 2011 17:41:50 +0100
Marek Kierdelewicz <marek@piasta.pl> wrote:

> Hi,
> 
> >... and since the packets are not encapsulated yet, how they will
> >return from ifb to vlanX to be encapsulated??
> >I see no tc command doing this in you example in your previous post.
> >Is there a missing command?
> 
> That's the way ifb works. Example of ingress shaping that uses "action
> mirred egress redirect":
> http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb#Typical_Usage
> 

Done. 

Working.

Thanx everyone.

Ethy

Re: shaping vlans - revisited

[Jan Engelhardt]

On Thursday 2011-02-03 21:01, Ethy H. Brito wrote:
>
>The marks is performed by these four lines:
>
>/usr/sbin/iptables -t mangle -A POSTROUTING -o eth0 -m mark ! --mark 0 \
>	-j IPMARK --addr dst --and-mask 0xffff --or-mask 0x20000
>/usr/sbin/iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 0 \
>	-j IPMARK --addr dst --and-mask 0xffff --or-mask 0x0
>/usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark ! --mark 2 \
>	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x0
>/usr/sbin/iptables -t mangle -A POSTROUTING -o eth1 -m mark --mark 2 \
>	-j IPMARK --addr src --and-mask 0xffff --or-mask 0x40000
>
>BTW "mark 2" is there in the iptable LOG.
>IN=eth0 OUT=vlan3 SRC=...blablabla ... ACK PSH URGP=0 MARK=0x2
>
>But where can I find IPMARK??

The IPMARK target controls the nfmark (MARK= field).

Re: shaping vlans - revisited

[Grant Taylor]

On 2/3/2011 2:01 PM, Ethy H. Brito wrote:
> One extra detail is: this setup was working (just as it is for eth0)
> until two weeks ago when then we created the vlans and splited the
> traffic among them.

Once the traffic is in side of a VLAN tag (on the raw device), filters 
will not match the traffic the say way that it did.  -  Try applying 
your filters to the VLAN sub-interface.

This is because the traffic is no longer IP traffic in side of Ethernet 
SNAP frames, but rather VLAN tagged frames that contain Ethernet SNAP 
frames.

Are you wanting to give one VLAN on a given raw interface more priority 
than another?



Grant. . . .

Re: shaping vlans - revisited

[Ethy H. Brito]

On Thu, 03 Feb 2011 21:42:59 -0600
Grant Taylor <gtaylor@riverviewtech.net> wrote:

> On 2/3/2011 2:01 PM, Ethy H. Brito wrote:  
> > One extra detail is: this setup was working (just as it is for eth0)
> > until two weeks ago when then we created the vlans and splited the
> > traffic among them.  
> 
> Once the traffic is in side of a VLAN tag (on the raw device), filters 
> will not match the traffic the say way that it did.  -  Try applying 
> your filters to the VLAN sub-interface.  

you mean: 

tc filter add dev vlanA ... match mark client1
tc filter add dev vlanB ... match mark client1
tc filter add dev vlanC ... match mark client1
...
 
for each client?!?!?! the will be hundreds!!

Doesn´t it imply having "tc {class,qdisc} add dev vlan[A-Z]" or " 
tc {class,qdisc} add dev eth1" will do?

> 
> This is because the traffic is no longer IP traffic in side of Ethernet 
> SNAP frames, but rather VLAN tagged frames that contain Ethernet SNAP 
> frames.
> 
> Are you wanting to give one VLAN on a given raw interface more priority 
> than another?  

Nope. The vlans are to other ISPs. Each ISP demands 2 Vlans (IPv4+IPv6).
I need to shape global traffic from/to each client no matter which ISP
he/she is using.

Ethy

Re: shaping vlans - revisited

[Ethy H. Brito]

On Thu, 03 Feb 2011 21:42:59 -0600
Grant Taylor <gtaylor@riverviewtech.net> wrote:

> Once the traffic is in side of a VLAN tag (on the raw device), filters 
> will not match the traffic the say way that it did.  -  Try applying 
> your filters to the VLAN sub-interface.
> 
> This is because the traffic is no longer IP traffic in side of Ethernet 
> SNAP frames, but rather VLAN tagged frames that contain Ethernet SNAP 
> frames.

Is that a way to use instead of "protocol ip" use something like
"protocol 0x8100"??

What is the general form of "protocol ip" (if there is one)?
Some "u32 match" perhaps??

Ethy