Not understand some in htb_do_events function

Not understand some in htb_do_events function

[Badalian Vyacheslav]

Hello all.
I have many messages like "htb: too many events !" in dmesg.

Try to see code and find that function try do 500 events at call.
Hm... may anyone ask why 500? Why its not dynamic value based on 
performance of PC?

Thanks for answers.

Re: Not understand some in htb_do_events function

[Patrick McHardy]

Badalian Vyacheslav wrote:
> Hello all.
> I have many messages like "htb: too many events !" in dmesg.
> 
> Try to see code and find that function try do 500 events at call.
> Hm... may anyone ask why 500? Why its not dynamic value based on 
> performance of PC?


Thats a good question, I wonder why it is limited at all.
Martin, any hints?

Re: Not understand some in htb_do_events function

[Martin Devera]

Patrick McHardy wrote:
> Badalian Vyacheslav wrote:
>> Hello all.
>> I have many messages like "htb: too many events !" in dmesg.
>>
>> Try to see code and find that function try do 500 events at call.
>> Hm... may anyone ask why 500? Why its not dynamic value based on 
>> performance of PC?
> 
> 
> Thats a good question, I wonder why it is limited at all.
> Martin, any hints?
> 
> 

Hi, I recently replied someone to the same question:

 > it is possible when during one jiffie (1 or 10ms) more than 500 classes
 > changed its state. It is meant to protect your system from livelock.
 > The constant should be set to something like
 > bogomips/bogocomplexity_of_state_change but it was not done.

the solution I have in my mind is to change
    if (net_ratelimit())
                  printk(KERN_WARNING "htb: too many events !\n");
          return HZ/10;
  to
    return 1;

to drain extra events asap. It the time of writing I was not able to
come with better solution and there were more bugs related to this
part of code than now.
We want way to smooth big burst of events over more dequeue invocations
in order to not slow dequeue too much. Constant 500 is max. allowed
"slowdown" of dequeue.
Any bright idea how to do it more elegant, Patrick ?

brgds, devik

Re: Not understand some in htb_do_events function

[Badalian Vyacheslav]

Martin Devera пишет:
> Patrick McHardy wrote:
>> Badalian Vyacheslav wrote:
>>> Hello all.
>>> I have many messages like "htb: too many events !" in dmesg.
>>>
>>> Try to see code and find that function try do 500 events at call.
>>> Hm... may anyone ask why 500? Why its not dynamic value based on 
>>> performance of PC?
>>
>>
>> Thats a good question, I wonder why it is limited at all.
>> Martin, any hints?
>>
>>
>
> Hi, I recently replied someone to the same question:
>
> > it is possible when during one jiffie (1 or 10ms) more than 500 classes
> > changed its state. It is meant to protect your system from livelock.
> > The constant should be set to something like
> > bogomips/bogocomplexity_of_state_change but it was not done.
>
> the solution I have in my mind is to change
>    if (net_ratelimit())
>                  printk(KERN_WARNING "htb: too many events !\n");
>          return HZ/10;
>  to
>    return 1;
>
> to drain extra events asap. It the time of writing I was not able to
> come with better solution and there were more bugs related to this
> part of code than now.
> We want way to smooth big burst of events over more dequeue invocations
> in order to not slow dequeue too much. Constant 500 is max. allowed
> "slowdown" of dequeue.
> Any bright idea how to do it more elegant, Patrick ?
>
> brgds, devik

Hmm.. i simple see that if i get messages like this in dmesg my PC limit 
speed, but i have many free CPU and HTB classes have not full bandwidth 
load....
If i delete half rules - i get bandwidth that i wont. I wrong understand 
that if we have this limit (constant 500) my PC can't get full performance?

Look scheme (example in my memory... not real numbers... for idea only):
HTB (default 1)
- Class 1 (rate 250mbs cell 500 prio 1)
---- Here 8000 HTB rules (parent 1) (rate 1kbs, cell from 200kbs to 5mbs)
- Class 2 (rate 250mbs cell 500 prio 1)

I always go to 2.
try ttcp.
current rate in class 1 - above 150mbs
current rate in class 2 - above 200mbs
Load avg above 0.10 - 0.25
Idle - 80%
 
Try delete half rules
current rate in class 1 - above 100mbs
current rate in class 2 - above 400mbs
Load avg above 0.10 - 0.25
Idle - 80%

All traffic in class 2 do i witch ttcp! its not deleted clients from 
class 1.

Any idea? Thanks.

Re: Not understand some in htb_do_events function

[Martin Devera]

Badalian Vyacheslav wrote:
> Martin Devera пишет:
>> Patrick McHardy wrote:
>>> Badalian Vyacheslav wrote:
>>>> Hello all.
>>>> I have many messages like "htb: too many events !" in dmesg.
>>>>
>>>> Try to see code and find that function try do 500 events at call.
>>>> Hm... may anyone ask why 500? Why its not dynamic value based on 
>>>> performance of PC?
>>>
>>>
>>> Thats a good question, I wonder why it is limited at all.
>>> Martin, any hints?
>>>
>>>
>>
>> Hi, I recently replied someone to the same question:
>>
>> > it is possible when during one jiffie (1 or 10ms) more than 500 classes
>> > changed its state. It is meant to protect your system from livelock.
>> > The constant should be set to something like
>> > bogomips/bogocomplexity_of_state_change but it was not done.
>>
>> the solution I have in my mind is to change
>>    if (net_ratelimit())
>>                  printk(KERN_WARNING "htb: too many events !\n");
>>          return HZ/10;
>>  to
>>    return 1;
>>
>> to drain extra events asap. It the time of writing I was not able to
>> come with better solution and there were more bugs related to this
>> part of code than now.
>> We want way to smooth big burst of events over more dequeue invocations
>> in order to not slow dequeue too much. Constant 500 is max. allowed
>> "slowdown" of dequeue.
>> Any bright idea how to do it more elegant, Patrick ?
>>
>> brgds, devik
> 
> Hmm.. i simple see that if i get messages like this in dmesg my PC limit 
> speed, but i have many free CPU and HTB classes have not full bandwidth 
> load....
> If i delete half rules - i get bandwidth that i wont. I wrong understand 
> that if we have this limit (constant 500) my PC can't get full performance?
> 
> Look scheme (example in my memory... not real numbers... for idea only):
> HTB (default 1)
> - Class 1 (rate 250mbs cell 500 prio 1)
> ---- Here 8000 HTB rules (parent 1) (rate 1kbs, cell from 200kbs to 5mbs)
> - Class 2 (rate 250mbs cell 500 prio 1)
> 
> I always go to 2.
> try ttcp.
> current rate in class 1 - above 150mbs
> current rate in class 2 - above 200mbs
> Load avg above 0.10 - 0.25
> Idle - 80%
> 
> Try delete half rules
> current rate in class 1 - above 100mbs
> current rate in class 2 - above 400mbs
> Load avg above 0.10 - 0.25
> Idle - 80%
> 
> All traffic in class 2 do i witch ttcp! its not deleted clients from 
> class 1.
> 
> Any idea? Thanks.

The change above (returning 1 instead of HZ/10) should help, it was tested. Also
you CAN increase the 500 to say 2000. I didn't the change as I'm not sure it is
good generic fix...

Re: Not understand some in htb_do_events function

[Patrick McHardy]

Martin Devera wrote:
> Patrick McHardy wrote:
>> Badalian Vyacheslav wrote:
>>> Hello all.
>>> I have many messages like "htb: too many events !" in dmesg.
>>>
>>> Try to see code and find that function try do 500 events at call.
>>> Hm... may anyone ask why 500? Why its not dynamic value based on 
>>> performance of PC?
>>
>>
>> Thats a good question, I wonder why it is limited at all.
>> Martin, any hints?
>>
> Hi, I recently replied someone to the same question:
> 
>  > it is possible when during one jiffie (1 or 10ms) more than 500 classes
>  > changed its state. It is meant to protect your system from livelock.
>  > The constant should be set to something like
>  > bogomips/bogocomplexity_of_state_change but it was not done.
> 
> the solution I have in my mind is to change
>    if (net_ratelimit())
>                  printk(KERN_WARNING "htb: too many events !\n");
>          return HZ/10;
>  to
>    return 1;
> 
> to drain extra events asap. It the time of writing I was not able to
> come with better solution and there were more bugs related to this
> part of code than now.

So this was meant to protect against endless loops?

> We want way to smooth big burst of events over more dequeue invocations
> in order to not slow dequeue too much. Constant 500 is max. allowed
> "slowdown" of dequeue.
> Any bright idea how to do it more elegant, Patrick ?


Unfortunately not, but I believe simply removing the limit
completely would be better than picking an arbitary value.

Re: Not understand some in htb_do_events function

[Martin Devera]

> 
> So this was meant to protect against endless loops?
> 
>> We want way to smooth big burst of events over more dequeue invocations
>> in order to not slow dequeue too much. Constant 500 is max. allowed
>> "slowdown" of dequeue.
>> Any bright idea how to do it more elegant, Patrick ?
> 
> 
> Unfortunately not, but I believe simply removing the limit
> completely would be better than picking an arbitary value.

Grrr my comp crashed while I was writing this mail. Well the second
attempt.
When we allow unlimited events per dequeue, then there is case where
all N classes in qdisc can be in the event queue with the same target
time. Then they will all be acted on in the loop within single dequeue,
costing us say some milliseconds. Additionaly, it tends to repeat itself
then in cycles.
Maybe it is acceptable but it seemed to me as rather big latency.
Thus I wanted to do only limited work per dequeue call. One possibility
is to remove the limit and "see what happend in wild".

What do u think about to do limited no of transitions and then schedule
tasklet to do the rest (again in limited buckets) ?

Re: Not understand some in htb_do_events function

[Patrick McHardy]

Martin Devera wrote:
>>
>> So this was meant to protect against endless loops?
>>
>>> We want way to smooth big burst of events over more dequeue invocations
>>> in order to not slow dequeue too much. Constant 500 is max. allowed
>>> "slowdown" of dequeue.
>>> Any bright idea how to do it more elegant, Patrick ?
>>
>>
>> Unfortunately not, but I believe simply removing the limit
>> completely would be better than picking an arbitary value.
> 
> Grrr my comp crashed while I was writing this mail. Well the second
> attempt.
> When we allow unlimited events per dequeue, then there is case where
> all N classes in qdisc can be in the event queue with the same target
> time. Then they will all be acted on in the loop within single dequeue,
> costing us say some milliseconds. Additionaly, it tends to repeat itself
> then in cycles.

I see.

> Maybe it is acceptable but it seemed to me as rather big latency.
> Thus I wanted to do only limited work per dequeue call. One possibility
> is to remove the limit and "see what happend in wild".
> 
> What do u think about to do limited no of transitions and then schedule
> tasklet to do the rest (again in limited buckets) ?


Alternatively we could just remove the printk and do what you
suggested first, return q->now + 1 to immediately continue
processing, but send out a packet first.

Re: Not understand some in htb_do_events function

[David Miller]

From: Patrick McHardy <kaber@trash.net>
Date: Tue, 15 Jan 2008 16:54:41 +0100

> Martin Devera wrote:
> > to drain extra events asap. It the time of writing I was not able to
> > come with better solution and there were more bugs related to this
> > part of code than now.
> 
> So this was meant to protect against endless loops?

I think, as Martin tried to explain further, it's trying
to avoid cases where the amount of work is extremely large.

Heuristics like this are by definition going to hit cases
where the value chosen is wrong or inappropriate.  On fast
cpus maybe a value of 50000 instead of 500 would work "best"

Therefore I would suggest removing the limit altogether for now and in
the long term work on whatever deficiencies in the algorithm make this
expensive enough to warrant limits in the first place.