[LARTC] HFSC and prioritization

[LARTC] HFSC and prioritization

[Eliot, Wireless and Server Administrator, Great Lakes Internet]

I'm using HFSC to limit bandwidth for our wireless customers. However,
I'd also like the bandwidth prioritized based on packet type. This is
what I'm trying right now, and I'd just like some input from anyone out
there knowledgeable in this on whether it does what I want it to do:

Eth1 -> HFSC 
........|-> HFSC (User1) (Min 512 Kb, Max 1024 Kb, Burst 1536 Kb@2s)
........|...|-> Prio_qdisc
........|.......|-> 1
........|.......|...|->HFSC VoIP/Interactive(max 30ms, Real 128Kb)
........|.......|
........|.......|-> 2
........|.......|...|->HFSC Web,FTP(Min 512Kb,Max 1024Kb,Burst
1536Kb@2s)
........|.......|
........|.......|-> 3
........|...........|->HFSC P2P(Min 0 Kb, Max 1024 Kb, all shared)
........|
........|-> HFSC (User2) (etc)
..etc...

What I'm aiming for is:

No matter what type of traffic is being transferred, the user is
guaranteed 512Kbps and can max out at 1024Kbps, but can also burst up to
1536Kbps for 2 seconds. 

VoIP / SSH / Telnet / non-data ACK packets get priority over everything
else. It would be guaranteed 128Kbps of bandwidth if it were needed.
Ideally, it would not reserve that bandwidth unless it was actually
needed. It should also get more bandwidth than 128Kbps if it is needed,
but anything after 128Kbps it has to fight for with everything else
(except it has a higher priority so it should win out in 99.99999% of
cases?) 

Web, FTP, and other unclassifiable traffic would get second priority and
would be guaranteed 512Kbps of bandwidth with a maximum of 1024Kbps and
can burst up to 1536Kbps for 2 seconds.

Finally, P2P traffic is not guaranteed anything, but could use all
available bandwidth if the user is not doing anything else. No matter
how much bandwidth P2P wants, if something else needs bandwidth, the
other traffic should win out and receive the bandwidth.


These are my actual rules:

# Base user class
tc class add dev wivl4 parent 5:0 classid 5:130 hfsc ls m1 1536.0Kbit d
2000ms m2 512.00Kbit ul m2 1024Kbit

# Priority queue
tc qdisc add dev wivl4 parent 5:130 handle 134:  prio bands 3
tc qdisc add dev wivl4 parent 134:1 handle 135:  hfsc default 1
tc qdisc add dev wivl4 parent 134:2 handle 136:  hfsc default 1
tc qdisc add dev wivl4 parent 134:3 handle 137:  hfsc default 1

# VoIP / Interactive
tc class add dev wivl4 parent 135: classid 135:1 hfsc sc umax 1500b dmax
30ms rate 128Kbit

# Web
tc class add dev wivl4 parent 136: classid 136:1 hfsc rt m2 512.00Kbit
ls m2 512.0Kbit ul m2 1024Kbit

# P2P
tc class add dev wivl4 parent 137: classid 137:1 hfsc ls m2 1024.0Kbit
ul m2 1024.0Kbit


Does this look appropriate?


Eliot Gable
Certified Wireless Network Administrator (CWNA)
Certified Wireless Security Professional (CWSP)
Cisco Certified Network Associate (CCNA)
CompTIA Security+ Certified
CompTIA Network+ Certified
Network and Systems Administrator
Great Lakes Internet, Inc.
112 North Howard
Croswell, MI 48422
(810) 679-3395
(877) 558-8324
 
Now offering Broadband Wireless Internet access in Croswell, Lexington,
Brown City, Yale, and Sandusky. Call for details.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Jody Shumaker]

> # Base user class
> tc class add dev wivl4 parent 5:0 classid 5:130 hfsc ls m1 1536.0Kbit d
> 2000ms m2 512.00Kbit ul m2 1024Kbit
>
> # Priority queue
> tc qdisc add dev wivl4 parent 5:130 handle 134:  prio bands 3
> tc qdisc add dev wivl4 parent 134:1 handle 135:  hfsc default 1
> tc qdisc add dev wivl4 parent 134:2 handle 136:  hfsc default 1
> tc qdisc add dev wivl4 parent 134:3 handle 137:  hfsc default 1
>
> # VoIP / Interactive
> tc class add dev wivl4 parent 135: classid 135:1 hfsc sc umax 1500b dmax
> 30ms rate 128Kbit
>
> # Web
> tc class add dev wivl4 parent 136: classid 136:1 hfsc rt m2 512.00Kbit
> ls m2 512.0Kbit ul m2 1024Kbit
>
> # P2P
> tc class add dev wivl4 parent 137: classid 137:1 hfsc ls m2 1024.0Kbit
> ul m2 1024.0Kbit
>
>
> Does this look appropriate?
>
>

My understanding of HFSC is limited, but i'm fairly sure its similar
to all other qdiscs in one respect that would make the config you have
shown, not actually work as you've described. Each of those HFSC
qdiscs is a seperate entity, no sharing will occur between those HFSC
classes because each one belongs to a different qdisc. If you
implemented this,  the priority portion would likely work, but if all
3 classes were sending they would be trying for their individual max,
resulting in a combined total send of 3072kbps.  If you want what
you've described then there would need to be some sort of way to
define that same priority within one HSFC between the classes.  I know
HTB has the prio parameter but I've never found good documentation on
HSFC and don't know if it has an equivalent.

Of course all of what I said would be wrong if HSFC has inter-qdisc
communication but I highly doubt that as it seems to go against the
design of most qdiscs.

What is there for good HSFC documentation out there right now anyways?

Thanks,
Jody
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Patrick McHardy]

Jody Shumaker wrote:
> My understanding of HFSC is limited, but i'm fairly sure its similar
> to all other qdiscs in one respect that would make the config you have
> shown, not actually work as you've described. Each of those HFSC
> qdiscs is a seperate entity, no sharing will occur between those HFSC
> classes because each one belongs to a different qdisc. If you
> implemented this,  the priority portion would likely work, but if all
> 3 classes were sending they would be trying for their individual max,
> resulting in a combined total send of 3072kbps.

Correct.

> If you want what
> you've described then there would need to be some sort of way to
> define that same priority within one HSFC between the classes.  I know
> HTB has the prio parameter but I've never found good documentation on
> HSFC and don't know if it has an equivalent.

HFSC doesn't support strict priorities (and neither does HTB, the
priorities just affect unused bandwidth and is still limited by the
ceiling). At least in the case of HFSC this is intentional, strict
priority is not very friendly because it allows traffic to be
entirely excluded, HFSC's goals are to enable flexible sharing by
allowing to seperately specify bandwidth and delay requirements.
If you really want strict priority you can use the prio qdisc as
_child_ of HFSC.

> Of course all of what I said would be wrong if HSFC has inter-qdisc
> communication but I highly doubt that as it seems to go against the
> design of most qdiscs.
> 
> What is there for good HSFC documentation out there right now anyways?

There is the original papers by Hui Zhang et al., which is mostly
about the theory and not very suitable for users - but still worth
reading if you're not scared by use of some math.
There used to be some documentation called "HFSC for Router Plugins",
which is partially applicable for Linux .. and some ALTQ and *BSD
documentation which is partially applicable as well. Besides that
there seem to be a few german student research projects about this
subject, but all I know of are in german. Last thing I know of is
an article written by a friend of mine for the german Linux Magazin,
unfortunately also only in german, but reviewed by myself and mostly
correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are
welcome :)
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Patrick McHardy]

Patrick McHardy wrote:
> Jody Shumaker wrote:
> 
>>What is there for good HSFC documentation out there right now anyways?
> 
> 
> There is the original papers by Hui Zhang et al., which is mostly
> about the theory and not very suitable for users - but still worth
> reading if you're not scared by use of some math.
> There used to be some documentation called "HFSC for Router Plugins",
> which is partially applicable for Linux .. and some ALTQ and *BSD
> documentation which is partially applicable as well. Besides that
> there seem to be a few german student research projects about this
> subject, but all I know of are in german. Last thing I know of is
> an article written by a friend of mine for the german Linux Magazin,
> unfortunately also only in german, but reviewed by myself and mostly
> correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are
> welcome :)

I forgot to mention, there is actually a quite easy way to get
started with HFSC if you already have a working HTB setup. Just
delete the "prio" statements, replace "htb" by "hfsc",
"rate" by "sc rate", "ceil" by "ul rate" and delete all the
remaining htb specific parameters and you have something working.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Patrick McHardy]

Alexandru Dragoi wrote:

> I think i'd like more docs in english about hfsc.

Me too. I don't have time to write one myself (and I'm not good at
this), but I can assist if anyone wants to do it.


> I would like to know
> also some tips about scalability at large amount of traffic, like more
> than 100mbit and more than 20kpps. I had a setup one that share 200mbit
> on 2 imq devices(both with a parent class of 200mbit), each with about
> 1000 hfsc classes. About 2 classes with only rt courves, another 2 with
> rt and ls and ul courves, and the rest (end users) with ls and ul
> courves. All only with m2 parametter. And at high traffic packet loss
> appeared. After switching to htb, no more packet loss.
> 
> Thanks in advance.


Mhh .. I know of setups where HFSC is running with 10k classes and high
bandwidth (>= 100mbit, don't know the exact amount). When switchting to
rbtrees I made some benchmarks and it performed almost identical to HTB,
so my guess is that its related to IMQ, which still seems to be pretty
broken. It could of course also be a different configuration mistake,
hard to tell without seeing the actual configuration.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Andy Furniss]

Patrick McHardy wrote:
> Patrick McHardy wrote:
> 
>>Jody Shumaker wrote:
>>
>>
>>>What is there for good HSFC documentation out there right now anyways?
>>
>>
>>There is the original papers by Hui Zhang et al., which is mostly
>>about the theory and not very suitable for users - but still worth
>>reading if you're not scared by use of some math.
>>There used to be some documentation called "HFSC for Router Plugins",
>>which is partially applicable for Linux .. and some ALTQ and *BSD
>>documentation which is partially applicable as well. Besides that
>>there seem to be a few german student research projects about this
>>subject, but all I know of are in german. Last thing I know of is
>>an article written by a friend of mine for the german Linux Magazin,
>>unfortunately also only in german, but reviewed by myself and mostly
>>correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are
>>welcome :)
> 
> 
> I forgot to mention, there is actually a quite easy way to get
> started with HFSC if you already have a working HTB setup. Just
> delete the "prio" statements, replace "htb" by "hfsc",
> "rate" by "sc rate", "ceil" by "ul rate" and delete all the
> remaining htb specific parameters and you have something working.

One thing to note is that HFSC will drop, rather than pass unshaped, 
traffic that is unclassified.

So if you don't use a default class and don't filter arp to a class then 
HFSC will appear broken whereas HTB will work.

Andy.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Jody Shumaker]

> HFSC doesn't support strict priorities (and neither does HTB, the
> priorities just affect unused bandwidth and is still limited by the
> ceiling). At least in the case of HFSC this is intentional, strict
> priority is not very friendly because it allows traffic to be
> entirely excluded, HFSC's goals are to enable flexible sharing by
> allowing to seperately specify bandwidth and delay requirements.
> If you really want strict priority you can use the prio qdisc as
> _child_ of HFSC.
>

I always forget this about the prio and HTB.  With HSFC does use of
the the max latency settings possibly get the desired goal from using
prio? I think this is what always appealed to me about HSFC from the
little I could understand.  That if I had an interactive class, it'd
always favor getting those packets through sooner than others, trying
to honor a latency, if I set it up correctly.

> > What is there for good HSFC documentation out there right now anyways?
>
> There is the original papers by Hui Zhang et al., which is mostly
> about the theory and not very suitable for users - but still worth
> reading if you're not scared by use of some math.
> There used to be some documentation called "HFSC for Router Plugins",
> which is partially applicable for Linux .. and some ALTQ and *BSD
> documentation which is partially applicable as well. Besides that
> there seem to be a few german student research projects about this
> subject, but all I know of are in german. Last thing I know of is
> an article written by a friend of mine for the german Linux Magazin,
> unfortunately also only in german, but reviewed by myself and mostly
> correct (klaus.geekserver.net/hfsc/hfsc.html) - translations are
> welcome :)
>

Unfortunately the original papers were a bit too confusing for me when
I last read them.  I could understand parts of it, but translating
that into hsfc options I would actually want to use, I couldn't do.
This german article looks like it'd have a lot of the clerification
I'd need, but a babelfish translation was still too difficult to
understand.   If someone's up for writing a good translation I'd
greatly appreciate it.

- Jody
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Patrick McHardy]

Andy Furniss wrote:
> One thing to note is that HFSC will drop, rather than pass unshaped,
> traffic that is unclassified.
> 
> So if you don't use a default class and don't filter arp to a class then
> HFSC will appear broken whereas HTB will work.

Good point, that is a trap that might easily make it appear as if HFSC
is stalling under some conditions.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Patrick McHardy]

Jody Shumaker wrote:
>> HFSC doesn't support strict priorities (and neither does HTB, the
>> priorities just affect unused bandwidth and is still limited by the
>> ceiling). At least in the case of HFSC this is intentional, strict
>> priority is not very friendly because it allows traffic to be
>> entirely excluded, HFSC's goals are to enable flexible sharing by
>> allowing to seperately specify bandwidth and delay requirements.
>> If you really want strict priority you can use the prio qdisc as
>> _child_ of HFSC.
>>
> 
> I always forget this about the prio and HTB.  With HSFC does use of
> the the max latency settings possibly get the desired goal from using
> prio? I think this is what always appealed to me about HSFC from the
> little I could understand.  That if I had an interactive class, it'd
> always favor getting those packets through sooner than others, trying
> to honor a latency, if I set it up correctly.


Exactly. I think strict priority is mostly used because of laziness,
unknown conditions or unflexible algorithms. You almost never want
one kind of traffic to be able to stall everything else (which BTW
raises some doubts about Linux's default choice of a 3band prio qdisc).
HFSC solves one and a half of these problems: without seperated
bandwidth/delay specifications you are unable to express that some
traffic should get good delay but doesn't need much guaranteed
bandwidth, so you have to use priorities. The half solved problem
is unknown conditions, it can also work in work-conserving mode,
which means that it will work fine on wireless or similar networks
or without any maximum bandwidth specification, but in that case
it can only provide fair sharing, no absolute guarantees. Only
half-solved because the unknown condition could also be the amount
of bandwidth or the delay required, in which case strict priority
might be the only viable option.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] HFSC and prioritization

[Alexandru Dragoi]

Patrick McHardy wrote:

>Alexandru Dragoi wrote:
>
>  
>
>>I think i'd like more docs in english about hfsc.
>>    
>>
>
>Me too. I don't have time to write one myself (and I'm not good at
>this), but I can assist if anyone wants to do it.
>
>
>  
>
>>I would like to know
>>also some tips about scalability at large amount of traffic, like more
>>than 100mbit and more than 20kpps. I had a setup one that share 200mbit
>>on 2 imq devices(both with a parent class of 200mbit), each with about
>>1000 hfsc classes. About 2 classes with only rt courves, another 2 with
>>rt and ls and ul courves, and the rest (end users) with ls and ul
>>courves. All only with m2 parametter. And at high traffic packet loss
>>appeared. After switching to htb, no more packet loss.
>>
>>Thanks in advance.
>>    
>>
>
>
>Mhh .. I know of setups where HFSC is running with 10k classes and high
>bandwidth (>= 100mbit, don't know the exact amount). When switchting to
>rbtrees I made some benchmarks and it performed almost identical to HTB,
>so my guess is that its related to IMQ, which still seems to be pretty
>broken. It could of course also be a different configuration mistake,
>hard to tell without seeing the actual configuration.
>_______________________________________________
>LARTC mailing list
>LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>  
>
Hello, here are some lines of the hfsc script i used.

#!/bin/bash

tc=/sbin/tc

$tc qdisc  add dev imq0 root handle 1: hfsc default 3
$tc class add dev imq0 parent 1:  classid 1:3 hfsc ls m2 200mbit ul m2
200mbit
$tc class add dev imq0 parent 1:  classid 1:2 hfsc ls m2 200mbit ul m2
200mbit


$tc qdisc  add dev imq1 root handle 1: hfsc default 3
$tc class  add dev imq1 parent 1:  classid 1:3 hfsc ls m2 20mbit ul m2
20mbit
$tc class  add dev imq1 parent 1:  classid 1:2 hfsc ls m2 20mbit ul m2
20mbit

$tc qdisc  add dev imq2 root handle 1: hfsc default 3
$tc class  add dev imq2 parent 1:  classid 1:3 hfsc ls m2 200mbit ul m2
200mbit
$tc class  add dev imq2 parent 1:  classid 1:2 hfsc ls m2 200mbit ul m2
200mbit

$tc qdisc  add dev imq3 root handle 1: hfsc default 3
$tc class  add dev imq3 parent 1:  classid 1:3 hfsc ls m2 20mbit ul m2
20mbit
$tc class  add dev imq3 parent 1:  classid 1:2 hfsc ls m2 20mbit ul m2
20mbit


## An important client

$tc class add dev imq0 parent 1:2 classid 1:0x6031 hfsc rt m2 100mbit
$tc qdisc add dev imq0 parent 1:0x6031 sfq
$tc filter add dev imq0 parent 1: protocol ip prio 10 u32 match ip dst
x.y.z.0/22 flowid 1:0x6031

$tc class add dev imq1 parent 1:2 classid 1:0x6031 hfsc rt m2 9mbit
$tc qdisc add dev imq1 parent 1:0x6031 sfq
$tc filter add dev imq1 parent 1: protocol ip prio 10 u32 match ip dst
x.y.z.0/22 flowid 1:0x6031

$tc class add dev imq2 parent 1:2 classid 1:0x6031 hfsc rt m2 100mbit
$tc qdisc add dev imq2 parent 1:0x6031 sfq
$tc filter add dev imq2 parent 1: protocol ip prio 10 u32 match ip src
x.y.z.0/22 flowid 1:0x6031

$tc class add dev imq3 parent 1:2 classid 1:0x6031 hfsc rt m2 9mbit
$tc qdisc add dev imq3 parent 1:0x6031 sfq
$tc filter add dev imq3 parent 1: protocol ip prio 10 u32 match ip src
x.y.z.0/22 flowid 1:0x6031

There were also a client with rt m2 20mbit ls m2 20mbit ul m2 100mbit on
imq0 and im2, then rt m2 5mbit ls m2 5mbit ul m2 10mbit on other 2 imqs.

The rest of clients , arround 1000, has each something like:

$tc class add dev imq0 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul
m2 20480Kbit
$tc qdisc add dev imq0 parent 1:0x116a sfq
$tc filter add dev imq0 parent 1: protocol ip prio 10 u32 match ip dst
a.b.c.d/32 flowid 1:0x116a

$tc class add dev imq1 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul
m2 256Kbit
$tc qdisc add dev imq1 parent 1:0x116a sfq
$tc filter add dev imq1 parent 1: protocol ip prio 10 u32 match ip dst
a.b.c.d/32 flowid 1:0x116a

$tc class add dev imq2 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul
m2 20480Kbit
$tc qdisc add dev imq2 parent 1:0x116a sfq
$tc filter add dev imq2 parent 1: protocol ip prio 10 u32 match ip src
a.b.c.d/32 flowid 1:0x116a

$tc class add dev imq3 parent 1:2 classid 1:0x116a hfsc ls m2 16Kbit ul
m2 256Kbit
$tc qdisc add dev imq3 parent 1:0x116a sfq
$tc filter add dev imq3 parent 1: protocol ip prio 10 u32 match ip src
a.b.c.d/32 flowid 1:0x116a


Some of them has rates half or double than the numbers above, depending
how much they pay.
Last lines of script are:

$tc class change dev imq0 parent 1:  classid 1:3 hfsc ls m2 16Kbit ul m2
256Kbit
$tc class change dev imq1 parent 1:  classid 1:3 hfsc ls m2 8Kbit ul m2
128Kbit
$tc class change dev imq2 parent 1:  classid 1:3 hfsc ls m2 16Kbit ul m2
256Kbit
$tc class change dev imq3 parent 1:  classid 1:3 hfsc ls m2 8Kbit ul m2
128Kbit

Now, the machine has multiple interfaces, i think 2 gigabit cards with
about 5-6 vlans. The ideea of imqs was to shape only traffic that comes
or goes from or to some vlans, traffic between clients being unshaped.
The machine also did bgp with 1400 prefixes learned and default route,
everything running on an 2.6.11 kernel, i think. I'm sure there was a
need for lots of tuning, like u32 hash filters .. and many others. With
this setup on high traffic things went crazy, like packet loss, real
time classes also didn;t get theyr traffic and so on. But ONLY changing
from hfsc to a htb, things worked much better, and the important clients
got theyr guatanteed bandwidth. Since then .. things changed a lot, u32
hash filters are used, with htb, i get a job somewhere else and so on.
:) Any ideas about what i did there are really welcome. Thank You.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc