bandwidth-limiting on LAN interface egress (2)

bandwidth-limiting on LAN interface egress (2)

[Lloyd Standish]

I have improved my previous post in hope of some advice, or at least a suggestion on where to ask this sort of question.

Suppose one is building a netfilter router, LAN to Internet, with multiple outward-facing interfaces (eth1, eth2, and eth3). There needs to be load-balancing over the outward interfaces.  There needs to be bandwidth-limiting for users on the LAN.  Users are typical Internet users (primarily http download with some important interactive traffic such as VOIP.)

Theoretically, can per-user bandwidth-limiting be done on egress of the LAN using htb+prio+sfq without encountering insurmountable latency problems due to queuing of incoming packets in the router?  Should traffic shaping (prioritizing of packets for interactive traffic) probably be an adequate solution to any latency problems?

Is there a way to use a policing queuing discipline in a case like this? (I assume it would have to be on egress of the LAN interface, since I cannot see how to police on ingress of the Internet-facing interfaces due to the per-user bandwidth-limiting.)

-- 
Lloyd

Re: bandwidth-limiting on LAN interface egress (2)

[Andrew Beverley]

On Wed, 2011-11-16 at 09:50 -0600, Lloyd Standish wrote:
> I have improved my previous post in hope of some advice, or at least a
>  suggestion on where to ask this sort of question.

Yep, I am planning on reading your other post when I get round to it,
but it was quite long...

> Suppose one is building a netfilter router, LAN to Internet, with
>  multiple outward-facing interfaces (eth1, eth2, and eth3). There needs
>  to be load-balancing over the outward interfaces.

Have you got this part working okay?

>  There needs to be
>  bandwidth-limiting for users on the LAN.  Users are typical Internet
>  users (primarily http download with some important interactive traffic
>  such as VOIP.)
> 
> Theoretically, can per-user bandwidth-limiting be done on egress of the
>  LAN using htb+prio+sfq without encountering insurmountable latency
>  problems due to queuing of incoming packets in the router?

If you are limiting per-user, then depending on the number of users, you
may run into problems. There was a similar discussion here a while ago:

http://comments.gmane.org/gmane.comp.security.firewalls.netfilter.general/41664

The upshot of that thread was to switch to u32 hashing filters if you
can.

>   Should
>  traffic shaping (prioritizing of packets for interactive traffic)
>  probably be an adequate solution to any latency problems?

I do it reasonably successfully with a lot of users on a small link.
http://andybev.com has some of the details.

> Is there a way to use a policing queuing discipline in a case like
>  this?

My personal opinion is that you shouldn't limit per user. You should
instead prioritise traffic properly. This way you'll have a lot less
classes and a lot less overhead. The traffic of heavy users over-using
the link will get less priority than low-bandwidth applications, so you
will achieve the same effect.

>  (I assume it would have to be on egress of the LAN interface,
>  since I cannot see how to police on ingress of the Internet-facing
>  interfaces due to the per-user bandwidth-limiting.)

Correct, you need to stick with egress shaping, so your inbound links
from the internet will need to be shaped on the internal LAN interface.

Andy

Re: bandwidth-limiting on LAN interface egress (2)

[Lloyd Standish]

On Wed, 16 Nov 2011 16:00:03 -0600, Andrew Beverley <andy@andybev.com> wrote:

>
>> Suppose one is building a netfilter router, LAN to Internet, with
>>  multiple outward-facing interfaces (eth1, eth2, and eth3). There needs
>>  to be load-balancing over the outward interfaces.
>
> Have you got this part working okay?

Thanks very much for your comments (below). Yes, load-balancing is working.  Also, preliminary traffic-shaping of interactive traffic is working as well as the per-user bandwidth limiting, via u32 hashing filters.  My earlier post has a lot of the bash code, which I posted earlier mainly because I thought it might be of use to someone doing something similar.

>
>>  There needs to be
>>  bandwidth-limiting for users on the LAN.  Users are typical Internet
>>  users (primarily http download with some important interactive traffic
>>  such as VOIP.)
>>
>> Theoretically, can per-user bandwidth-limiting be done on egress of the
>>  LAN using htb+prio+sfq without encountering insurmountable latency
>>  problems due to queuing of incoming packets in the router?
>
> If you are limiting per-user, then depending on the number of users, you
> may run into problems. There was a similar discussion here a while ago:
>
> http://comments.gmane.org/gmane.comp.security.firewalls.netfilter.general/41664
>
> The upshot of that thread was to switch to u32 hashing filters if you
> can.

I saw that thread a while back...  it was the only place I found mention of use of u32 hashing filters for the IFB device.  After seeing that I was also able to use the IFB device to limit upload speed (LAN ingress).

Since I used the u32 hashing filters I think there should be no performance penalty due to iterating lots of rules, even if my friend get hundreds of users.  I even got the hashing filters working "multilevel" - that is, for lookup of the subnet portion of the IP address, but that is currently not in use since he only has a single subnet.

>>   Should traffic shaping (prioritizing of packets for interactive traffic)
>>  probably be an adequate solution to any latency problems?
>
> I do it reasonably successfully with a lot of users on a small link.
> http://andybev.com has some of the details.

Wow, what you have done looks cool.  I will study your configuration, thanks!
>
>> Is there a way to use a policing queuing discipline in a case like
>>  this?
>
> My personal opinion is that you shouldn't limit per user. You should
> instead prioritise traffic properly. This way you'll have a lot less
> classes and a lot less overhead. The traffic of heavy users over-using
> the link will get less priority than low-bandwidth applications, so you
> will achieve the same effect.
>
I agree in principle, but my friend needs the per-user bandwidth limiting.  Actually, there are only about 5 bandwidth classes into which all users fall (by LAN IP), and with the hashing filters it looks pretty "lean and mean" to my inexperienced eye.  The bash code that creates the tc commands is in my previous post.

>>  (I assume it would have to be on egress of the LAN interface,
>>  since I cannot see how to police on ingress of the Internet-facing
>>  interfaces due to the per-user bandwidth-limiting.)
>
> Correct, you need to stick with egress shaping, so your inbound links
> from the internet will need to be shaped on the internal LAN interface.

Can I assume from your comments that you think queuing in the router should not cause big latency problems as long as interactive traffic is given highest priority?  If so, I will go ahead and install this firewall and start testing it under some simulated loads.

Again, thanks for the advice!

Re: bandwidth-limiting on LAN interface egress (2)

[Lloyd Standish]

I was under the impression that one could define an HTB qdisc, then set up bandwidth classes for of it, then use u32 filters matching on LAN IP numbers to assign the flows for several IPs to the same HTB class.  I tried that and found that the total bandwidth of the class is shared among the connections, which is not what I wanted or expected!

Apparently for bandwidth-limiting to work there needs to be a HTB class declared for *each* user IP number!  This certainly makes sense if the sum of classes' rates should total the rate of the containing HTB.

Andy, you wrote:
"My personal opinion is that you shouldn't limit per user. You should
instead prioritise traffic properly. This way you'll have a lot less
classes and a lot less overhead."

Now I understand what you mean by "a lot less classes"!

Unless someone corrects me, I will assume that there is no way to use filters to assign several "flows" (by user IP) to the same class, such that each flow gets the class's full configured rate.

This leads me to another question: Consider the case where there are 50 users, each with 1 Mbit download rate, but statistics show that all 50 *never* require the full bandwidth, and that a total 35 Mbit total connection rate is sufficient.  Should in this case the HTB qdisc be configured to have total 35 Mbit (or slightly less), even though the sum of all classes' rates will be 50 Mbit?

This might answer my question, or it is an error on the lartc site:

http://lartc.org/howto/lartc.qdisc.classful.html#AEN903 shows an example in which the sum of classes' rates is higher than the total HTB qdisc rate:

"9.5.5.1. Sample configuration

Functionally almost identical to the CBQ sample configuration above:

# tc qdisc add dev eth0 root handle 1: htb default 30

# tc class add dev eth0 parent 1: classid 1:1 htb rate 6mbit burst 15k

# tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5mbit burst 15k
# tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit ceil 6mbit burst 15k
# tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1kbit ceil 6mbit burst 15k"

Thanks in advance for any advice.

Re: bandwidth-limiting on LAN interface egress (2)

[Andrew Beverley]

On Wed, 2011-11-16 at 23:06 -0600, Lloyd Standish wrote:
> I was under the impression that one could define an HTB qdisc, then set
>  up bandwidth classes for of it, then use u32 filters matching on LAN
>  IP numbers to assign the flows for several IPs to the same HTB class. 
>  I tried that and found that the total bandwidth of the class is shared
>  among the connections, which is not what I wanted or expected!
> 
> Apparently for bandwidth-limiting to work there needs to be a HTB class
>  declared for *each* user IP number!  This certainly makes sense if the
>  sum of classes' rates should total the rate of the containing HTB.
> 
> Andy, you wrote:
> "My personal opinion is that you shouldn't limit per user. You should
> instead prioritise traffic properly. This way you'll have a lot less
> classes and a lot less overhead."
> 
> Now I understand what you mean by "a lot less classes"!
> 
> Unless someone corrects me, I will assume that there is no way to use
>  filters to assign several "flows" (by user IP) to the same class, such
>  that each flow gets the class's full configured rate.

No, I don't think there is any other way. You can use the "flow"
parameter to balance per user IP within a class, but you won't be able
to use that to hard-limit to a particular bandwidth for the IP, just the
overall rate limit of the class:

$TC filter add dev eth0 parent 30: protocol ip \
        handle 1 flow hash keys dst divisor 1024

> This leads me to another question: Consider the case where there are 50
>  users, each with 1 Mbit download rate, but statistics show that all 50
>  *never* require the full bandwidth, and that a total 35 Mbit total
>  connection rate is sufficient.  Should in this case the HTB qdisc be
>  configured to have total 35 Mbit (or slightly less), even though the
>  sum of all classes' rates will be 50 Mbit?

The way to achieve this is to set a root limit of 35 Mbit, then make
sure that all the "rate" values add up to that. You can then set a
"ceil" value to allow the leaf classes to burst up to 1 Mbit.

> 
> This might answer my question, or it is an error on the lartc site:
> 
> http://lartc.org/howto/lartc.qdisc.classful.html#AEN903 shows an
>  example in which the sum of classes' rates is higher than the total
>  HTB qdisc rate:

I personally think that is a mistake. Everything I have read in the past
states that the leaf classes should add up to the total root rate.

Andy

Re: bandwidth-limiting on LAN interface egress (2)

[Andrew Beverley]

On Wed, 2011-11-16 at 16:56 -0600, Lloyd Standish wrote:
> Can I assume from your comments that you think queuing in the router
>  should not cause big latency problems as long as interactive traffic
>  is given highest priority?

My personal opinion is that it should be okay, although depends what you
define as "big latency". I've been running the rules previously
described for a while now, and whilst they're not perfect, the latency
is minimal (I comfortably use SSH whilst the link is saturated).

You might also want to check out some of the "buffer bloat" discussions:

http://netoptimizer.blogspot.com/

http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-mastermind-bufferbloat/

Andy

Re: bandwidth-limiting on LAN interface egress (2)

[Lloyd Standish]

On Sun, 20 Nov 2011 07:38:39 -0600, Andrew Beverley <andy@andybev.com> wrote:

> On Wed, 2011-11-16 at 16:56 -0600, Lloyd Standish wrote:
>> Can I assume from your comments that you think queuing in the router
>>  should not cause big latency problems as long as interactive traffic
>>  is given highest priority?
>
> My personal opinion is that it should be okay, although depends what you
> define as "big latency". I've been running the rules previously
> described for a while now, and whilst they're not perfect, the latency
> is minimal (I comfortably use SSH whilst the link is saturated).
>
> You might also want to check out some of the "buffer bloat" discussions:
>
> http://netoptimizer.blogspot.com/
>
> http://gettys.wordpress.com/2010/12/03/introducing-the-criminal-mastermind-bufferbloat/
>
> Andy
>

Hi Andy,
Thanks much for your insights, and for the "buffer bloat" links.

We will be implementing per-user bandwidth-limiting in a couple of weeks.  However, for the moment I was (urgently) asked to skip the bandwidth-limiting and just get a load-balancing router going with some traffic shaping.  That allowed me to stop using qdiscs on the LAN interface.  I have an HTB qdisc on egress of each outward-facing interface, and a policing qdisc on each ingress.

It appears to be load-balancing OK, using "nexthop" to multiple routing tables in the default route, but I can't set a default route in each of the "custom" routing tables.  (See my other thread.)

--
Lloyd