Packetlost when "tc qdisc del dev eth0 root"

Packetlost when "tc qdisc del dev eth0 root"

[Badalian Vyacheslav]

Hello all. Have packetlost when do "tc qdisc del dev eth0 root".

look:

slavon ~ # ping -f 87.255.1.134
PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data.
.
.....................................................................................................................................................................................................................................................................
.
--- 87.255.1.134 ping statistics ---
60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms
rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, 
ipg/ewma 0.650/2.197 ms

Its normal?
Maybe if tc do changes - need create second queue (hash of rules or how 
you named it?) and do changes at it. Then replace old queue rules by 
created new.
Logic -
1. Do snapshot
2. Do changes in shapshot
3. All new packets go to snapshot
4. If old queue not have packets - delete it.
5. Snapshot its default.

Thanks for answers.
Slavon.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Patrick McHardy]

Badalian Vyacheslav wrote:
> Hello all. Have packetlost when do "tc qdisc del dev eth0 root".
> 
> look:
> 
> slavon ~ # ping -f 87.255.1.134
> PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data.
> .
> ..................................................................................................................................................................................................................................................................... 
> 
> .
> --- 87.255.1.134 ping statistics ---
> 60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms
> rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219, 
> ipg/ewma 0.650/2.197 ms
> 
> Its normal?

Yes, packets in the old qdisc are lost.

> Maybe if tc do changes - need create second queue (hash of rules or how 
> you named it?) and do changes at it. Then replace old queue rules by 
> created new.
> Logic -
> 1. Do snapshot
> 2. Do changes in shapshot
> 3. All new packets go to snapshot
> 4. If old queue not have packets - delete it.
> 5. Snapshot its default.


That doesn't really work since qdiscs keep internal state that
in large parts depends on the packets queued. Take the qlen as
a simple example, the new qdisc doesn't know about the packets
in the old one and will exceed the limit.

Re: Packetlost when "tc qdisc del dev eth0 root"

[slavon]

I understand. Thanks!

> Badalian Vyacheslav wrote:
>> Hello all. Have packetlost when do "tc qdisc del dev eth0 root".
>>
>> look:
>>
>> slavon ~ # ping -f 87.255.1.134
>> PING 87.255.1.134 (87.255.1.134) 56(84) bytes of data.
>> .
>> .....................................................................................................................................................................................................................................................................   
>> .
>> --- 87.255.1.134 ping statistics ---
>> 60811 packets transmitted, 60544 received, 0% packet loss, time 39528ms
>> rtt min/avg/max/mdev = 0.203/0.579/3227.655/13.124 ms, pipe 219,   
>> ipg/ewma 0.650/2.197 ms
>>
>> Its normal?
>
> Yes, packets in the old qdisc are lost.
>
>> Maybe if tc do changes - need create second queue (hash of rules or  
>>  how you named it?) and do changes at it. Then replace old queue   
>> rules by created new.
>> Logic -
>> 1. Do snapshot
>> 2. Do changes in shapshot
>> 3. All new packets go to snapshot
>> 4. If old queue not have packets - delete it.
>> 5. Snapshot its default.
>
>
> That doesn't really work since qdiscs keep internal state that
> in large parts depends on the packets queued. Take the qlen as
> a simple example, the new qdisc doesn't know about the packets
> in the old one and will exceed the limit.



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Jarek Poplawski]

Patrick McHardy wrote, On 01/15/2008 05:05 PM:

> Badalian Vyacheslav wrote:

...

> Yes, packets in the old qdisc are lost.
> 
>> Maybe if tc do changes - need create second queue (hash of rules or how 
>> you named it?) and do changes at it. Then replace old queue rules by 
>> created new.
>> Logic -
>> 1. Do snapshot
>> 2. Do changes in shapshot
>> 3. All new packets go to snapshot
>> 4. If old queue not have packets - delete it.
>> 5. Snapshot its default.
> 
> 
> That doesn't really work since qdiscs keep internal state that
> in large parts depends on the packets queued. Take the qlen as
> a simple example, the new qdisc doesn't know about the packets
> in the old one and will exceed the limit.

But, some similar alternative to killing packets 'to death' could
be imagined, I suppose (in the future, of course!). So, e.g. doing
the switch automatically after last packet has been dequeued (maybe
even with some 'special' function/mode for this). After all even
with accuracy lost, it could be less visible for clients than
current way?

Regards,
Jarek P.

Re: Packetlost when "tc qdisc del dev eth0 root"

[slavon]

Quoting Jarek Poplawski <jarkao2@gmail.com>:

> Patrick McHardy wrote, On 01/15/2008 05:05 PM:
>
>> Badalian Vyacheslav wrote:
>
> ...
>
>> Yes, packets in the old qdisc are lost.
>>
>>> Maybe if tc do changes - need create second queue (hash of rules or how
>>> you named it?) and do changes at it. Then replace old queue rules by
>>> created new.
>>> Logic -
>>> 1. Do snapshot
>>> 2. Do changes in shapshot
>>> 3. All new packets go to snapshot
>>> 4. If old queue not have packets - delete it.
>>> 5. Snapshot its default.
>>
>>
>> That doesn't really work since qdiscs keep internal state that
>> in large parts depends on the packets queued. Take the qlen as
>> a simple example, the new qdisc doesn't know about the packets
>> in the old one and will exceed the limit.
>
> But, some similar alternative to killing packets 'to death' could
> be imagined, I suppose (in the future, of course!). So, e.g. doing
> the switch automatically after last packet has been dequeued (maybe
> even with some 'special' function/mode for this). After all even
> with accuracy lost, it could be less visible for clients than
> current way?
>
> Regards,
> Jarek P.

Hmmm... i found way to fix this for me... but its not look good

Scheme look like:
Root - prio bands 3 priomap 0 0 0 0 ....
--- Class 1
--- Class 2
-------- Copy of all table (Last this qdisc be root)
--- Class 3
-------- Copy of all table (Last this qdisc be root)

2. Add filter to root - flowid all packets to class 2
3. Delete qdisc at class 3
4. Create all table on class 3 (~20k qdiscs and 20k classes)
5. Replace filter on root - flowid all packets to class 3
6. If need update go to step 3, but use class 2

All work good... and packets not dropeed =)
But i have above 45 k classes and qdiscs.... After some time i will  
need patch to up max qdisc and classes more then 65k (> 0xfffe) =)))
Also i have very bad TC commands performance then i have more then 10k rules.

Thanks =)

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: Packetlost when "tc qdisc del dev eth0 root"

[slavon]

Good night! =)

Sorry... i was wrong...
I see that problem more serious....

Lets see to scheme

Class 1
---qdisc
------- 10k classes
Class 2
---qdisc
------- 10k classes

All traffic go to class 2... class 1 qdisc not have packets and if we  
delete it - packets not lost... in theory... lets try delete class 1  
qdisc (all childrens delete too)...
PC freeze on 2-5 seconds... its not forward any traffic at this  
moment... its great tree lock?

Its normal or code need to more accurate lock?

Thanks!

> Quoting Jarek Poplawski <jarkao2@gmail.com>:
>
>> Patrick McHardy wrote, On 01/15/2008 05:05 PM:
>>
>>> Badalian Vyacheslav wrote:
>>
>> ...
>>
>>> Yes, packets in the old qdisc are lost.
>>>
>>>> Maybe if tc do changes - need create second queue (hash of rules or how
>>>> you named it?) and do changes at it. Then replace old queue rules by
>>>> created new.
>>>> Logic -
>>>> 1. Do snapshot
>>>> 2. Do changes in shapshot
>>>> 3. All new packets go to snapshot
>>>> 4. If old queue not have packets - delete it.
>>>> 5. Snapshot its default.
>>>
>>>
>>> That doesn't really work since qdiscs keep internal state that
>>> in large parts depends on the packets queued. Take the qlen as
>>> a simple example, the new qdisc doesn't know about the packets
>>> in the old one and will exceed the limit.
>>
>> But, some similar alternative to killing packets 'to death' could
>> be imagined, I suppose (in the future, of course!). So, e.g. doing
>> the switch automatically after last packet has been dequeued (maybe
>> even with some 'special' function/mode for this). After all even
>> with accuracy lost, it could be less visible for clients than
>> current way?
>>
>> Regards,
>> Jarek P.
>
> Hmmm... i found way to fix this for me... but its not look good
>
> Scheme look like:
> Root - prio bands 3 priomap 0 0 0 0 ....
> --- Class 1
> --- Class 2
> -------- Copy of all table (Last this qdisc be root)
> --- Class 3
> -------- Copy of all table (Last this qdisc be root)
>
> 2. Add filter to root - flowid all packets to class 2
> 3. Delete qdisc at class 3
> 4. Create all table on class 3 (~20k qdiscs and 20k classes)
> 5. Replace filter on root - flowid all packets to class 3
> 6. If need update go to step 3, but use class 2
>
> All work good... and packets not dropeed =)
> But i have above 45 k classes and qdiscs.... After some time i will
> need patch to up max qdisc and classes more then 65k (> 0xfffe) =)))
> Also i have very bad TC commands performance then i have more then 10k rules.
>
> Thanks =)
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.



----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Jarek Poplawski]

On Wed, Jan 16, 2008 at 01:04:59AM +0300, slavon@bigtelecom.ru wrote:
> Good night! =)

Good morning! ;)

>
> Sorry... i was wrong...
> I see that problem more serious....
>
> Lets see to scheme
>
> Class 1
> ---qdisc
> ------- 10k classes
> Class 2
> ---qdisc
> ------- 10k classes
>
> All traffic go to class 2... class 1 qdisc not have packets and if we 
> delete it - packets not lost... in theory... lets try delete class 1 qdisc 
> (all childrens delete too)...
> PC freeze on 2-5 seconds... its not forward any traffic at this moment... 
> its great tree lock?
>
> Its normal or code need to more accurate lock?

I don't think it's normal. On the other hand I've never had such
problems... Probably this all isn't optimized for such big operations,
so maybe you should try to do this with some smaller chunks?

Jarek P.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Patrick McHardy]

slavon@bigtelecom.ru wrote:
> Good night! =)
> 
> Sorry... i was wrong...
> I see that problem more serious....
> 
> Lets see to scheme
> 
> Class 1
> ---qdisc
> ------- 10k classes
> Class 2
> ---qdisc
> ------- 10k classes
> 
> All traffic go to class 2... class 1 qdisc not have packets and if we 
> delete it - packets not lost... in theory... lets try delete class 1 
> qdisc (all childrens delete too)...
> PC freeze on 2-5 seconds... its not forward any traffic at this 
> moment... its great tree lock?
> 
> Its normal or code need to more accurate lock?


htb class destruction can be quite expansive if one of the rb trees
needs to be rebalanced. Doing that for 10000 classes would explain
the delay.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Jarek Poplawski]

Patrick McHardy wrote, On 01/16/2008 06:12 AM:

> slavon@bigtelecom.ru wrote:
>> Good night! =)
>>
>> Sorry... i was wrong...
>> I see that problem more serious....
>>
>> Lets see to scheme
>>
>> Class 1
>> ---qdisc
>> ------- 10k classes
>> Class 2
>> ---qdisc
>> ------- 10k classes
>>
>> All traffic go to class 2... class 1 qdisc not have packets and if we 
>> delete it - packets not lost... in theory... lets try delete class 1 
>> qdisc (all childrens delete too)...
>> PC freeze on 2-5 seconds... its not forward any traffic at this 
>> moment... its great tree lock?
>>
>> Its normal or code need to more accurate lock?
> 
> 
> htb class destruction can be quite expansive if one of the rb trees
> needs to be rebalanced. Doing that for 10000 classes would explain
> the delay.

I've just looked at this and it seems there should be no rbtrees operations
during htb qdisc deletion (or I miss something).

Slavon, I see nearby that you teach oprofile...

Regards,
Jarek P.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Jarek Poplawski]

On Wed, Jan 16, 2008 at 12:46:02AM +0300, slavon@bigtelecom.ru wrote:
...
> Hmmm... i found way to fix this for me... but its not look good
>
> Scheme look like:
> Root - prio bands 3 priomap 0 0 0 0 ....
> --- Class 1
> --- Class 2
> -------- Copy of all table (Last this qdisc be root)
> --- Class 3
> -------- Copy of all table (Last this qdisc be root)
>
> 2. Add filter to root - flowid all packets to class 2
> 3. Delete qdisc at class 3
> 4. Create all table on class 3 (~20k qdiscs and 20k classes)
> 5. Replace filter on root - flowid all packets to class 3
> 6. If need update go to step 3, but use class 2
>
> All work good... and packets not dropeed =)
> But i have above 45 k classes and qdiscs.... After some time i will need 
> patch to up max qdisc and classes more then 65k (> 0xfffe) =)))
> Also i have very bad TC commands performance then i have more then 10k rules.

Right! I get your idea (not all details...), and this really shows
there is needed something simpler for this. 

Thanks,
Jarek P.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Glen Turner]

On Wed, 2008-01-16 at 00:46 +0300, slavon@bigtelecom.ru wrote:

> But i have above 45 k classes and qdiscs.... After some time i will  
> need patch to up max qdisc and classes more then 65k (> 0xfffe) =)))
> Also i have very bad TC commands performance then i have more then 10k rules.

In contrast a "brand name" router will support 4 to 16 queues
per (sub-)interface.  Your large number of queues exceeds
expectations.

What are you trying to do?  You may be better off inventing a
new qdisc to meet your need (eg, to do per-IP traffic shaping
or, less complexly, a traffic shaping based on the value of mark
which might offend DiffServ purists) or have, say, 1000 output
rates based on a marking and use the ipset feature of Netfilter
to set the mark.  Using 1000 rates gives an error of 0.1% which
is unlikely to be noticed by your customers given the larger
effects of shaping on TCP performance but is beneath the
level where you are noticing performance issues.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Patrick McHardy]

Jarek Poplawski wrote:
> Patrick McHardy wrote, On 01/15/2008 05:05 PM:
> 
>> Badalian Vyacheslav wrote:
> 
> ...
> 
>> Yes, packets in the old qdisc are lost.
>>
>>> Maybe if tc do changes - need create second queue (hash of rules or how 
>>> you named it?) and do changes at it. Then replace old queue rules by 
>>> created new.
>>> Logic -
>>> 1. Do snapshot
>>> 2. Do changes in shapshot
>>> 3. All new packets go to snapshot
>>> 4. If old queue not have packets - delete it.
>>> 5. Snapshot its default.
>>
>> That doesn't really work since qdiscs keep internal state that
>> in large parts depends on the packets queued. Take the qlen as
>> a simple example, the new qdisc doesn't know about the packets
>> in the old one and will exceed the limit.
> 
> But, some similar alternative to killing packets 'to death' could
> be imagined, I suppose (in the future, of course!). So, e.g. doing
> the switch automatically after last packet has been dequeued (maybe
> even with some 'special' function/mode for this). After all even
> with accuracy lost, it could be less visible for clients than
> current way?


This would need support from the qdiscs to do it properly. Looks
non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Jarek Poplawski]

On Wed, Jan 16, 2008 at 06:02:00AM +0100, Patrick McHardy wrote:
...
> This would need support from the qdiscs to do it properly. Looks
> non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.

Yes. At first I've thought this would need quite a lot of work, but
it seems, there could be probably used something very simple too,
like e.g. a 'dummy' sched switcher, which after replacing as a root
the old qdisc and knowing the pointer to the new one could simply
call for dequeing the old one and the new one for everything else.
Then, after completely dequeuing it would call destroy for the old
qdisc and probably switch itself with the new one as a root. If this
new one were created temporarily e.g. on a dummyX dev, and the switch
qdisc added to dummyY (as a temporary holder) with ethX and dummyX as
parameters, it seems this could be done without any API changes.
(But, of course, something more sophisticated should be even better.)

Regards,
Jarek P.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Patrick McHardy]

Jarek Poplawski wrote:
> On Wed, Jan 16, 2008 at 06:02:00AM +0100, Patrick McHardy wrote:
> ...
>> This would need support from the qdiscs to do it properly. Looks
>> non-trivial for HTB/HFSC/CBQ, but the others shouldn't be that hard.
> 
> Yes. At first I've thought this would need quite a lot of work, but
> it seems, there could be probably used something very simple too,
> like e.g. a 'dummy' sched switcher, which after replacing as a root
> the old qdisc and knowing the pointer to the new one could simply
> call for dequeing the old one and the new one for everything else.
> Then, after completely dequeuing it would call destroy for the old
> qdisc and probably switch itself with the new one as a root. If this
> new one were created temporarily e.g. on a dummyX dev, and the switch
> qdisc added to dummyY (as a temporary holder) with ethX and dummyX as
> parameters, it seems this could be done without any API changes.
> (But, of course, something more sophisticated should be even better.)


Yes, thats one possibility (without the dummy device though please).
But I wonder what this would actually be useful for. I don't think
replacing the root qdisc by a different type is a common scenario,
for the same type you can simply use "tc qdisc change", "tc class
change" and "tc class replace".

Badalian, what are you actually doing?

Re: Packetlost when "tc qdisc del dev eth0 root"

[Badalian Vyacheslav]

>
>
> Yes, thats one possibility (without the dummy device though please).
> But I wonder what this would actually be useful for. I don't think
> replacing the root qdisc by a different type is a common scenario,
> for the same type you can simply use "tc qdisc change", "tc class
> change" and "tc class replace".
>
> Badalian, what are you actually doing?
>
Sorry. Resend to all.

I simple recreate all rules. I change idea from do many 
add,change,delete because have many kernel panics on many kernels 2.6.x
First i have panics on "delete filter" operation... was fix it... 
great.. then have panics on "delete htb" operation... long time wait to 
fix it (maintainer of tc not have time to fix it i think).. but have 1-5 
panics at day.... i think my clients hate  me =) i rewrite script to 
simple  recreate all rules... in this method i have small packetlost 1 
time in hour (then recreate root qdisc), but not have panics... now i 
use new logic of scripts... =)
maybe need go back to accurate logic of scripts, but i to fear kernel 
panics =)

P.S. Feature request:
tc class show [ dev STRING ] [ root | parent CLASSID ]
may add classid filter to show?
i need do like this
/sbin/tc -s class show dev eth0 | grep -A2 "htb $HTB "
to get stats of class... but i every run real look all table.. its not good

Thanks that are you interesting history of my problems =)
Slavon.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Jarek Poplawski]

On Wed, Jan 16, 2008 at 11:35:35AM +0300, Badalian Vyacheslav wrote:
...
> I simple recreate all rules. I change idea from do many add,change,delete 
> because have many kernel panics on many kernels 2.6.x
> First i have panics on "delete filter" operation... was fix it... great.. 
> then have panics on "delete htb" operation... long time wait to fix it 
> (maintainer of tc not have time to fix it i think).. but have 1-5 panics at 
> day.... i think my clients hate  me =) i rewrite script to simple  recreate 
> all rules... in this method i have small packetlost 1 time in hour (then 
> recreate root qdisc), but not have panics... now i use new logic of 
> scripts... =)
> maybe need go back to accurate logic of scripts, but i to fear kernel 
> panics =)

BTW, I don't know about others, but it seems this bugzilla #9632 waits
for your testing, to find the reason of this bug... IMHO, if you can't
try this now, it's better to close this or/and at least add some
comment. And if there are any other bugs unfixed which are
reproducible and you can test (or have tested) some patches, please
resend them as new threads to the list. Alas, omitting the panics
can't help in removing these bugs.

Jarek P.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Jarek Poplawski]

On Wed, Jan 16, 2008 at 09:05:47AM +0100, Patrick McHardy wrote:
...
> Yes, thats one possibility (without the dummy device though please).
> But I wonder what this would actually be useful for. I don't think
> replacing the root qdisc by a different type is a common scenario,
> for the same type you can simply use "tc qdisc change", "tc class
> change" and "tc class replace".
>
> Badalian, what are you actually doing?

I'm not sure Vyacheslav needs just this, but I've thought about the
possibility to recreate the 'shadow' copy of currently used qdisc
tree (with some updates of course) while it's running. So, the
possibility of using all the same handles and classids, and even
dev names if possible, and doing such a switch without any visible
break.

Jarek P.

Re: Packetlost when "tc qdisc del dev eth0 root"

[Badalian Vyacheslav]

Jarek Poplawski пишет:
> On Wed, Jan 16, 2008 at 09:05:47AM +0100, Patrick McHardy wrote:
> ...
>   
>> Yes, thats one possibility (without the dummy device though please).
>> But I wonder what this would actually be useful for. I don't think
>> replacing the root qdisc by a different type is a common scenario,
>> for the same type you can simply use "tc qdisc change", "tc class
>> change" and "tc class replace".
>>
>> Badalian, what are you actually doing?
>>     
>
> I'm not sure Vyacheslav needs just this, but I've thought about the
> possibility to recreate the 'shadow' copy of currently used qdisc
> tree (with some updates of course) while it's running. So, the
> possibility of using all the same handles and classids, and even
> dev names if possible, and doing such a switch without any visible
> break.
>
> Jarek P.
>
>   
I also think that system must forward all packets what it get if it not 
dropped manual (by iptables or shaper).
Maybe someone need to test delete big TREE.. simple delete, not 
recreate... linux unavailable some time (if its realy big table its time 
may be 10-20 sec on 2xXeon).
I think need helper to do that operations more accurate. Now see 
situation that linux is PPTP server... its get 2000k connection... try 
delete qdisc on eth0 (incoming from wan to pptp clients)... i think many 
sessions will drop.

Thanks!