Linux 2.6 HTB does only work with existing connections

Linux 2.6 HTB does only work with existing connections

[Hua Zhong]

Hi all,

I tried to use HTB on a 2.6.12 based kernel. It seems that if I issue the tc
commands, existing connections adjust bandwidth immediately, but any new
connections won't be filtered correctly.

The tc version is 3.17.

The commands are very simple (only eth1 is active):

    tc qdisc del dev eth1 root
    tc qdisc add dev eth1 root handle 1: htb default 20
    tc class add dev eth1 parent 1: classid 1:20 htb rate 4000kbit

No filters, just limit everything to under 4Mbit.

Then run ttcp to a remote machine, ifconfig shows eth1 is not limited.

However, while ttcp is running, issue the above three commands again, the
bandwidth is then limited correctly.

Any ideas?

The following are the show commands:

# tc qdisc show dev eth1
qdisc htb 1: r2q 10 default 20 direct_packets_stat 11
# tc class show dev eth1
class htb 1:20 root prio 0 rate 4000Kbit ceil 4000Kbit burst 2099b cburst
2099b

Please cc me.

Hua

RE: Linux 2.6 HTB does only work with existing connections

[Hua Zhong]

Just add more info: this machine has TOE. And if that's turned off,
everything seems to work fine.

Does it mean that traffic control doesn't play well with TOE? Or there are
some tricks to make them work together?

Thanks for any pointers.

BTW: it's x86_64 but I guess it doesn't matter much.

> -----Original Message-----
> From: Hua Zhong [mailto:hzhong@gmail.com]
> Sent: Wednesday, February 21, 2007 5:45 PM
> To: 'netdev@vger.kernel.org'
> Subject: Linux 2.6 HTB does only work with existing connections
> 
> Hi all,
> 
> I tried to use HTB on a 2.6.12 based kernel. It seems that if I issue
> the tc commands, existing connections adjust bandwidth immediately, but
> any new connections won't be filtered correctly.
> 
> The tc version is 3.17.
> 
> The commands are very simple (only eth1 is active):
> 
>     tc qdisc del dev eth1 root
>     tc qdisc add dev eth1 root handle 1: htb default 20
>     tc class add dev eth1 parent 1: classid 1:20 htb rate 4000kbit
> 
> No filters, just limit everything to under 4Mbit.
> 
> Then run ttcp to a remote machine, ifconfig shows eth1 is not limited.
> 
> However, while ttcp is running, issue the above three commands again,
> the bandwidth is then limited correctly.
> 
> Any ideas?
> 
> The following are the show commands:
> 
> # tc qdisc show dev eth1
> qdisc htb 1: r2q 10 default 20 direct_packets_stat 11
> # tc class show dev eth1
> class htb 1:20 root prio 0 rate 4000Kbit ceil 4000Kbit burst 2099b
> cburst 2099b
> 
> Please cc me.
> 
> Hua

Re: Linux 2.6 HTB does only work with existing connections

[Andi Kleen]

"Hua Zhong" <hzhong@gmail.com> writes:

> Just add more info: this machine has TOE. And if that's turned off,
> everything seems to work fine.
> 
> Does it mean that traffic control doesn't play well with TOE? Or there are
> some tricks to make them work together?

Congratulations. You discovered one of the many reasons why TOE is a 
bad idea.

-Andi

RE: Linux 2.6 HTB does only work with existing connections

[Hua Zhong]

Correction: it's TSO, not TOE.

By googling I found that specifying a big MTU (like 16500) on the "tc class"
command avoids the "giant" packets. No idea where they come from.

Any better idea on a proper fix?

> -----Original Message-----
> From: Hua Zhong [mailto:hzhong@gmail.com]
> Sent: Wednesday, February 21, 2007 6:30 PM
> To: 'netdev@vger.kernel.org'
> Subject: RE: Linux 2.6 HTB does only work with existing connections
> 
> Just add more info: this machine has TOE. And if that's turned off,
> everything seems to work fine.
> 
> Does it mean that traffic control doesn't play well with TOE? Or there
> are some tricks to make them work together?
> 
> Thanks for any pointers.
> 
> BTW: it's x86_64 but I guess it doesn't matter much.
> 
> > -----Original Message-----
> > From: Hua Zhong [mailto:hzhong@gmail.com]
> > Sent: Wednesday, February 21, 2007 5:45 PM
> > To: 'netdev@vger.kernel.org'
> > Subject: Linux 2.6 HTB does only work with existing connections
> >
> > Hi all,
> >
> > I tried to use HTB on a 2.6.12 based kernel. It seems that if I issue
> > the tc commands, existing connections adjust bandwidth immediately,
> but
> > any new connections won't be filtered correctly.
> >
> > The tc version is 3.17.
> >
> > The commands are very simple (only eth1 is active):
> >
> >     tc qdisc del dev eth1 root
> >     tc qdisc add dev eth1 root handle 1: htb default 20
> >     tc class add dev eth1 parent 1: classid 1:20 htb rate 4000kbit
> >
> > No filters, just limit everything to under 4Mbit.
> >
> > Then run ttcp to a remote machine, ifconfig shows eth1 is not limited.
> >
> > However, while ttcp is running, issue the above three commands again,
> > the bandwidth is then limited correctly.
> >
> > Any ideas?
> >
> > The following are the show commands:
> >
> > # tc qdisc show dev eth1
> > qdisc htb 1: r2q 10 default 20 direct_packets_stat 11
> > # tc class show dev eth1
> > class htb 1:20 root prio 0 rate 4000Kbit ceil 4000Kbit burst 2099b
> > cburst 2099b
> >
> > Please cc me.
> >
> > Hua

Re: Linux 2.6 HTB does only work with existing connections

[Andi Kleen]

"Hua Zhong" <hzhong@gmail.com> writes:

> Correction: it's TSO, not TOE.
> 
> By googling I found that specifying a big MTU (like 16500) on the "tc class"
> command avoids the "giant" packets. No idea where they come from.

They come from TCP because it will pass down a max MTU sized packet
(upto 64k) for the NIC to segment (Tcp Segmentation Offload)

But the traffic control cannot work with such large packets.

> Any better idea on a proper fix?

Hmm, that's nasty.   The only workaround I could think of would
be to force TSO/LSO off when a non standard queueing discipline is active.

-Andi