Linux Traffic Shaping broken in 3.0.3rc1 ?

Linux Traffic Shaping broken in 3.0.3rc1 ?

[Timo Benk]

Hi,

with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it
worked very well and accurate with the HTB class:

Limit: 1mbit / actual rate: 116.2KB/s (given by scp)


Now i looked at 3.0.3rc1 and noticed that traffic shaping using the same
setup as before does not work as accurate as before (same hardware):

Limit: 1mbit / actual rate: 449.0KB/s (given by scp)


Interstingly shaping of the incoming transfer rate at the vif*.*
interfaces works correct.

The problem is reproducible on my side and for me it seems to be a
problem with the new Xen-kernel.

Here is a stripped down version of my tc setup for you to verify the
problem:

----<snip>----
tc qdisc add dev peth0 root handle 1:0 htb
tc class add dev peth0 parent 1:0 classid 1:1 htb rate 1mbit
tc qdisc add dev peth0 parent 1:1 handle 11: prio
tc filter add dev peth0 parent 1:0 protocol ip prio 1 u32 match ip src \
  172.17.0.100/32 flowid 1:1
----<snap>----

172.17.0.100/32 is the IP of my Domain-0.

Greetings,
-timo
-- 
Timo Benk - B1 Systems GmbH (http://www.b1-systems.de) 
Jabber ID: fry@jabber.org - ICQ ID: #241877854 
PGP Public Key: http://m28s01.vlinux.de/b1_gpg_key.asc

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Matt Ayres]

Timo Benk wrote:
> Hi,
> 
> with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it
> worked very well and accurate with the HTB class:
> 

Why not just use Xen's own rate limiting system that works directly on 
the vif device line?

Regards,
Matt

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Keir Fraser]

On 8/10/06 5:17 pm, "Matt Ayres" <matta@tektonic.net> wrote:

>> with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it
>> worked very well and accurate with the HTB class:
> 
> Why not just use Xen's own rate limiting system that works directly on
> the vif device line?

Yes, that's the best way. It's odd that TC behaviour has changed though.

 -- Keir

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Timo Benk]

Keir Fraser wrote:
> On 8/10/06 5:17 pm, "Matt Ayres" <matta@tektonic.net> wrote:
> 
>> with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it
>> worked very well and accurate with the HTB class:
>> Why not just use Xen's own rate limiting system that works directly on
>> the vif device line?
> 
> Yes, that's the best way. It's odd that TC behaviour has changed though.

AFAIK Xens rate limiting feature is limited to outgoing traffic and i
implement a rather complex (complex at least with tc) scenario:

- each domain will get a guaranteed minimum bandwidth
- if one domain is not using all of the guaranteed bandwidth, the not
  used bandwidth can be reused by other domains in addition to the
  guaranteed bandwidth they own
- in- and outcoming traffic will be limited

That is perfectly possible with tc and the HTB queue, but imho not with 
Xens rate limiting feature.

The other reason was that in my experience rate limiting in 3.0.2 was
buggy and leaving undestroyable zombie domains.

BTW a quick test on 3.0.3rc2 showed the following behaviour of Xen rate
limiting on my host:
---<snip>---
vif = [ 'rate=1MB/s' ]
---<snap>---

xendom0:~ # xm shutdown xendom2 # rate limited domain
[... wait until xendom2 shutdown finished ...]

xendom0:~ # xm list
Error: Device 0 not connected
Usage: xm list [options] [Domain, ...]

List information about all/some domains.
  -l, --long                     Output all VM details in SXP
  --label                        Include security labels


However, besides my personal problem with my setup that is not working
anymore i think tc is a important feature in QoS environments and should
work as solid on a Xen kernel as it runs on a legacy Linux kernel.

Greetings,
-timo

-- Timo Benk - B1 Systems GmbH (http://www.b1-systems.de) 
Jabber ID: fry@jabber.org - ICQ ID: #241877854 
PGP Public Key: http://m28s01.vlinux.de/b1_gpg_key.asc

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Keir Fraser]

On 8/10/06 19:30, "Timo Benk" <Timo.Benk@gmx.de> wrote:

> However, besides my personal problem with my setup that is not working
> anymore i think tc is a important feature in QoS environments and should
> work as solid on a Xen kernel as it runs on a legacy Linux kernel.

It sounds like maybe something has changed in the kernel config meaning that
your packets are no longer passing through the traffic classifier. Apart
from that, nothing has changed that should cause generic tc code to not work
properly any more.

 -- Keir

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Keir Fraser]

On 8/10/06 19:30, "Timo Benk" <Timo.Benk@gmx.de> wrote:

> BTW a quick test on 3.0.3rc2 showed the following behaviour of Xen rate
> limiting on my host:
> ---<snip>---
> vif = [ 'rate=1MB/s' ]
> ---<snap>---
> 
> xendom0:~ # xm shutdown xendom2 # rate limited domain
> [... wait until xendom2 shutdown finished ...]
> 
> xendom0:~ # xm list
> Error: Device 0 not connected
> Usage: xm list [options] [Domain, ...]

This is now fixed (the domain doesn't hang around as a zombie any more), but
the built-in rate limiting still seems broken (guest loses network
connectivity after a while). I'll have to investigate further.

As for save/restore of domains with non-dom0 backends. That's pushing the
envelope somewhat and might require some changes to our tools. But at the
very least it shouldn't leave zombies hanging around. I'll give your setup a
test and see what can be done.

 -- keir

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Keir Fraser]

On 9/10/06 17:18, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> As for save/restore of domains with non-dom0 backends. That's pushing the
> envelope somewhat and might require some changes to our tools. But at the
> very least it shouldn't leave zombies hanging around. I'll give your setup a
> test and see what can be done.

Actually, you cannot save/restore backend domains at all, even when they are
not actually configured to access real hardware. We should give a better
error message about that, of course. ;-)

The zombie xendom2 will exist until xendom1 is destroyed, for sure. This is
because xendom1 holds onto some xendom2 resources. *But* does xendom2
continue to exist as a zombie even after you forcibly 'xm destroy' xendom1?
That would definitely be a bug we want to fix.

 -- Keir

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Timo Benk]

Keir Fraser wrote:
> On 9/10/06 17:18, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>> As for save/restore of domains with non-dom0 backends. That's pushing the
>> envelope somewhat and might require some changes to our tools. But at the
>> very least it shouldn't leave zombies hanging around. I'll give your setup a
>> test and see what can be done.
> 
> Actually, you cannot save/restore backend domains at all, even when they are
> not actually configured to access real hardware. We should give a better
> error message about that, of course. ;-)
If it is not possible to save/restore a backend domain it is ok. The
reason why stumbled across this problem was that the xendomains script
tried to save the domains on reboot of Domain-0. That failed of course.

Then xendomains tries to shutdown these remaining domains. That fails
too and the host did not reboot for a long time. IMHO that is the main
problem.

The xen scripts that lie around in the system should be aware of the
fact that certain domains cannot be saved, and handle this situation
gracefully.

> The zombie xendom2 will exist until xendom1 is destroyed, for sure. This is
> because xendom1 holds onto some xendom2 resources. *But* does xendom2
> continue to exist as a zombie even after you forcibly 'xm destroy' xendom1?
You are right. After destroying xendom1 the zombie domain went away too.

Greetings,
-timo
-- 
Timo Benk - Jabber ID: fry@jabber.org - ICQ ID: #241877854
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Keir Fraser]

On 10/10/06 11:50, "Timo Benk" <timo.benk@gmx.de> wrote:

> The xen scripts that lie around in the system should be aware of the
> fact that certain domains cannot be saved, and handle this situation
> gracefully.

I think we'll add a feature flag that guest kernel can write to xenstore to
indicate whether they support save/restore. Alternatively the tools could
remember that they set up a guest as a backend of another. They probably do
that already in fact. Anyway, this is definitely fixable for 3.0.4.

 -- Keir

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Keir Fraser]

On 9/10/06 17:18, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

> This is now fixed (the domain doesn't hang around as a zombie any more), but
> the built-in rate limiting still seems broken (guest loses network
> connectivity after a while). I'll have to investigate further.

This is now fixed.

 -- Keir

Re: Linux Traffic Shaping broken in 3.0.3rc1 ?

[Keir Fraser]

On 7/10/06 20:21, "Timo Benk" <Timo.Benk@gmx.de> wrote:

> with Xen 3.0.2 i used tc to limit the outgoing traffic rate and it
> worked very well and accurate with the HTB class:
> Limit: 1mbit / actual rate: 116.2KB/s (given by scp)
>
> Now i looked at 3.0.3rc1 and noticed that traffic shaping using the same
> setup as before does not work as accurate as before (same hardware):
> Limit: 1mbit / actual rate: 449.0KB/s (given by scp)

This is not a Xen-specific bug. The problem is that now we support TSO,
packets are larger than the maximum that TC expects and so we overflow the
pre-computed rate table (and our 8kB packets get treated as 1.6kB packets).
To see this, run 'tc -s class show dev peth0'. You'll see a 'giants' field
that is non-zero -- those are all packets that were too big to handle
properly.

The fix is simple. Add an explicit mtu parameter to the htb class:
tc class add dev peth0 parent 1:0 classid 1:1 htb rate 1mbit mtu 16000

 Regards,
 Keir