Re: virtio GSO makes IPv6 unbearably slow

[Sridhar Samudrala <samudrala.sridhar@gmail.com>]

On 2/1/2010 11:50 AM, Bernhard Schmidt wrote:
> Hi,
>
> I have a really weird issue all of the sudden on _one_ of my two KVM
> hosts. The other one, while running on a different hardware in a
> different network, is configured in a very similar way and does not show
> these issues (so far).
>
> Host:
> - AMD Athlon64 3500+
> - Debian testing amd64
> - Kernel 2.6.32-trunk from Debian
> - Debian qemu-kvm 0.11.1+dfsg-1
>
> The system has a routed uplink (Intel e100) and an internal bridge, that
> connects all the VMs (only virtual ports). Networking in the VMs is
> usually configured like this:
>
> -net nic,model=virtio,macaddr=00:16:3E:7C:30:AA -net
> tap,ifname=vm.compile,script=no
>
> The guests are Debian amd64, either testing or stable, running a custom
> stripped 2.6.31.6 or 2.6.32.7. I've also tested older kernels (2.6.28.2
> and 2.6.30.4) and have seen the same problem.
>
> Problem:
> IPv6 tx throughput from a Guest to a host in the internet is extremely
> low (around 16kB/s). IPv6 rx is fine, as well as throughput in IPv4
> (both ways). Also fine is throughput from the guest to the host and from
> the host to the internet (also both ways).
>
> Running tcpdump on the tap-device on the host while doing an scp to
> another system shows this:
>
> 20:43:27.130438 IP6 GUEST.59864>  DEST.22: Flags [.], seq 70785:73641, ack 2128, win 327, options [nop,nop,TS val 4294920148 ecr 63373335], length 2856
> 20:43:27.130506 IP6 HOST>  GUEST: ICMP6, packet too big, mtu 1500, length 1240
> 20:43:27.131965 IP6 DEST.22>  GUEST.59864: Flags [.], ack 69357, win 501, options [nop,nop,TS val 63373335 ecr 4294920144], length 0
> 20:43:27.131996 IP6 DEST.22>  GUEST.59864: Flags [.], ack 70785, win 501, options [nop,nop,TS val 63373335 ecr 4294920144], length 0
> 20:43:27.132651 IP6 GUEST.59864>  DEST.22: Flags [.], seq 73641:76497, ack 2128, win 327, options [nop,nop,TS val 4294920148 ecr 63373335], length 2856
> 20:43:27.132704 IP6 HOST>  GUEST: ICMP6, packet too big, mtu 1500, length 1240
> 20:43:27.346347 IP6 GUEST.59864>  DEST.22: Flags [.], seq 70785:72213, ack 2128, win 327, options [nop,nop,TS val 4294920202 ecr 63373335], length 1428
> 20:43:27.360411 IP6 DEST.22>  GUEST.59864: Flags [.], ack 72213, win 501, options [nop,nop,TS val 63373358 ecr 4294920202], length 0
> 20:43:27.361045 IP6 GUEST.59864>  DEST.22: Flags [.], seq 76497:79353, ack 2128, win 327, options [nop,nop,TS val 4294920205 ecr 63373358], length 2856
>
> So, the guest sends packets of 2856 bytes, which are too large to pass
> through the eth0 of the host (1500 bytes). Thus, the host rejects the
> packet and sends back an ICMPv6 packet too big, which sometimes makes
> the guest send at least one packet with the small MTU, but sometimes it
> doesn't. This is repeated all over and slows down the process.
>
> According to ethtool GSO is enabled on the guest, but disabling it using
> ethtool does not have any effect. But giving the kernel
> "virtio_net.gso=0" in append fixes the issue completely.
>
> Anyone having any ideas?
>
>    
I ran into a similar problem when using Intel e1000e driver on the host. 
Found a bug in most of the intel ethernet drivers when handling large 
GSO IPv6 packets.
You can see the fix i submitted here
     
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8e1e8a4779cb23c1d9f51e9223795e07ec54d77a

But you seem to be using Intel e100 driver. So i am not sure you are 
running into the same issue.
You could try disabling both tso and gso on the guest using ethtool.

On the host, do you have gso/tso enabled on the uplink e100 driver?

Thanks
Sridhar