large divisor for flow classifier

large divisor for flow classifier

[Jonathan Thibault]

 It appears that when setting a fairly large divisor on the flow classifier for sfq, traffic stops altogether.

On my machine, anything above divisor 2200 seems to stop all traffic.  If I want to be fair between hosts (but not flows) for a large network (say 6000 hosts), I run into problems.  Obviously the rates here are quite low but this is just an example.

I also tested on a real interface with the same results.

Example that works:

tc qdisc add dev ifb0 root handle 3: htb default 10
  tc class add dev ifb0 parent 3: classid 3:1 htb rate 80kbit
    tc class add dev ifb0 parent 3:1 classid 3:10 htb rate 80kbit
      tc qdisc add dev ifb0 parent 3:10 handle 310: sfq perturb 10
      tc filter add dev ifb0 parent 3: protocol all prio 1 u32 match mark 0x000 0xf00 flowid 3:10
      tc filter add dev ifb0 parent 310: protocol all handle 0x310 flow hash keys dst divisor 1024

Example that doesn't work:

tc qdisc add dev ifb0 root handle 3: htb default 10
  tc class add dev ifb0 parent 3: classid 3:1 htb rate 80kbit
    tc class add dev ifb0 parent 3:1 classid 3:10 htb rate 80kbit
      tc qdisc add dev ifb0 parent 3:10 handle 310: sfq perturb 10
      tc filter add dev ifb0 parent 3: protocol all prio 1 u32 match mark 0x000 0xf00 flowid 3:10
      tc filter add dev ifb0 parent 310: protocol all handle 0x310 flow hash keys dst divisor 6144

Jonathan

Re: large divisor for flow classifier

[Eric Dumazet]

Le vendredi 15 octobre 2010 à 14:14 -0400, Jonathan Thibault a écrit :
> It appears that when setting a fairly large divisor on the flow classifier for sfq, traffic stops altogether.
> 
> On my machine, anything above divisor 2200 seems to stop all traffic.  If I want to be fair between hosts (but not flows) for a large network (say 6000 hosts), I run into problems.  Obviously the rates here are quite low but this is just an example.
> 
> I also tested on a real interface with the same results.
> 
> Example that works:
> 
> tc qdisc add dev ifb0 root handle 3: htb default 10
>   tc class add dev ifb0 parent 3: classid 3:1 htb rate 80kbit
>     tc class add dev ifb0 parent 3:1 classid 3:10 htb rate 80kbit
>       tc qdisc add dev ifb0 parent 3:10 handle 310: sfq perturb 10
>       tc filter add dev ifb0 parent 3: protocol all prio 1 u32 match mark 0x000 0xf00 flowid 3:10
>       tc filter add dev ifb0 parent 310: protocol all handle 0x310 flow hash keys dst divisor 1024
> 
> Example that doesn't work:
> 
> tc qdisc add dev ifb0 root handle 3: htb default 10
>   tc class add dev ifb0 parent 3: classid 3:1 htb rate 80kbit
>     tc class add dev ifb0 parent 3:1 classid 3:10 htb rate 80kbit
>       tc qdisc add dev ifb0 parent 3:10 handle 310: sfq perturb 10
>       tc filter add dev ifb0 parent 3: protocol all prio 1 u32 match mark 0x000 0xf00 flowid 3:10
>       tc filter add dev ifb0 parent 310: protocol all handle 0x310 flow hash keys dst divisor 6144
> 

SFQ is limited to a 1024 divisor

You might try following patch :

(8192 is the smallest power of two greater than 6144)

sizeof(struct sfq_sched_data) becomes 0x2ccc instead of 0x10cc

keep in mind hash distribution is not perfect.

What would be the real rate ?


diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
index 3cf478d..c4a53d6 100644
--- a/net/sched/sch_sfq.c
+++ b/net/sched/sch_sfq.c
@@ -77,7 +77,7 @@
 	It is easy to increase these values, but not in flight.  */
 
 #define SFQ_DEPTH		128
-#define SFQ_HASH_DIVISOR	1024
+#define SFQ_HASH_DIVISOR	8192
 
 /* This type should contain at least SFQ_DEPTH*2 values */
 typedef unsigned char sfq_index;

Re: large divisor for flow classifier

[Jarek Poplawski]

Eric Dumazet wrote:
> Le vendredi 15 octobre 2010 à 14:14 -0400, Jonathan Thibault a écrit :
>> It appears that when setting a fairly large divisor on the flow classifier for sfq, traffic stops altogether.
>>
>> On my machine, anything above divisor 2200 seems to stop all traffic.  If I want to be fair between hosts (but not flows) for a large network (say 6000 hosts), I run into problems.  Obviously the rates here are quite low but this is just an example.
...
> SFQ is limited to a 1024 divisor
> 
> You might try following patch :
> 
> (8192 is the smallest power of two greater than 6144)
> 
> sizeof(struct sfq_sched_data) becomes 0x2ccc instead of 0x10cc
> 
> keep in mind hash distribution is not perfect.
> 
> What would be the real rate ?
> 
> 
> diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
> index 3cf478d..c4a53d6 100644
> --- a/net/sched/sch_sfq.c
> +++ b/net/sched/sch_sfq.c
> @@ -77,7 +77,7 @@
>  	It is easy to increase these values, but not in flight.  */
>  
>  #define SFQ_DEPTH		128
> -#define SFQ_HASH_DIVISOR	1024
> +#define SFQ_HASH_DIVISOR	8192

Because of low SFQ_DEPTH, which limits its queue to 127 packets,
SFQ isn't suitable for serving so many users. There is sch_drr as
a replacement, alas more complex and undocumented, but google should
help you enough.

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=13d2a1d2b032de08d7dcab6a1edcd47802681f96

Jarek P.

Re: large divisor for flow classifier

[Jarek Poplawski]

Jarek Poplawski wrote:
> Because of low SFQ_DEPTH, which limits its queue to 127 packets,
> SFQ isn't suitable for serving so many users. There is sch_drr as
> a replacement, alas more complex and undocumented [...]

Sorry! Actually there is a man page already!

Kudos to Florian!
Jarek P.

Re: large divisor for flow classifier

[Jonathan Thibault]

I will certainly look into that.

Jonathan

On 15/10/10 05:10 PM, Jarek Poplawski wrote:
> Eric Dumazet wrote:
> 
> Because of low SFQ_DEPTH, which limits its queue to 127 packets,
> SFQ isn't suitable for serving so many users. There is sch_drr as
> a replacement, alas more complex and undocumented, but google should
> help you enough.
> 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=13d2a1d2b032de08d7dcab6a1edcd47802681f96
> 
> Jarek P.

Re: large divisor for flow classifier

[Jonathan Thibault]

Merci beaucoup :),

It at least shows I wasn't just confused about the way it works.  In its planned final form, the rate will be set around 175Mbit.  I don't need perfect distribution so things should be fine as long as hosts cannot easily cheat their way into having more bandwidth merely by creating more flows.

Jonathan

On 15/10/10 04:01 PM, Eric Dumazet wrote:
> Le vendredi 15 octobre 2010 à 14:14 -0400, Jonathan Thibault a écrit :
> 
> SFQ is limited to a 1024 divisor
> 
> You might try following patch :
> 
> (8192 is the smallest power of two greater than 6144)
> 
> sizeof(struct sfq_sched_data) becomes 0x2ccc instead of 0x10cc
> 
> keep in mind hash distribution is not perfect.
> 
> What would be the real rate ?
> 
> 
> diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
> index 3cf478d..c4a53d6 100644
> --- a/net/sched/sch_sfq.c
> +++ b/net/sched/sch_sfq.c
> @@ -77,7 +77,7 @@
>  	It is easy to increase these values, but not in flight.  */
>  
>  #define SFQ_DEPTH		128
> -#define SFQ_HASH_DIVISOR	1024
> +#define SFQ_HASH_DIVISOR	8192
>  
>  /* This type should contain at least SFQ_DEPTH*2 values */
>  typedef unsigned char sfq_index;
> 
>