From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sridhar Samudrala Subject: Re: virtio GSO makes IPv6 unbearably slow Date: Mon, 01 Feb 2010 22:08:53 -0800 Message-ID: <4B67C175.9030409@gmail.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Bernhard Schmidt Return-path: Received: from e32.co.us.ibm.com ([32.97.110.150]:43589 "EHLO e32.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752774Ab0BBGKF (ORCPT ); Tue, 2 Feb 2010 01:10:05 -0500 Received: from d03relay01.boulder.ibm.com (d03relay01.boulder.ibm.com [9.17.195.226]) by e32.co.us.ibm.com (8.14.3/8.13.1) with ESMTP id o1263sf8032080 for ; Mon, 1 Feb 2010 23:03:54 -0700 Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay01.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o126A0Qf135732 for ; Mon, 1 Feb 2010 23:10:00 -0700 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o1269xvA031783 for ; Mon, 1 Feb 2010 23:09:59 -0700 In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On 2/1/2010 11:50 AM, Bernhard Schmidt wrote: > Hi, > > I have a really weird issue all of the sudden on _one_ of my two KVM > hosts. The other one, while running on a different hardware in a > different network, is configured in a very similar way and does not show > these issues (so far). > > Host: > - AMD Athlon64 3500+ > - Debian testing amd64 > - Kernel 2.6.32-trunk from Debian > - Debian qemu-kvm 0.11.1+dfsg-1 > > The system has a routed uplink (Intel e100) and an internal bridge, that > connects all the VMs (only virtual ports). Networking in the VMs is > usually configured like this: > > -net nic,model=virtio,macaddr=00:16:3E:7C:30:AA -net > tap,ifname=vm.compile,script=no > > The guests are Debian amd64, either testing or stable, running a custom > stripped 2.6.31.6 or 2.6.32.7. I've also tested older kernels (2.6.28.2 > and 2.6.30.4) and have seen the same problem. > > Problem: > IPv6 tx throughput from a Guest to a host in the internet is extremely > low (around 16kB/s). IPv6 rx is fine, as well as throughput in IPv4 > (both ways). Also fine is throughput from the guest to the host and from > the host to the internet (also both ways). > > Running tcpdump on the tap-device on the host while doing an scp to > another system shows this: > > 20:43:27.130438 IP6 GUEST.59864> DEST.22: Flags [.], seq 70785:73641, ack 2128, win 327, options [nop,nop,TS val 4294920148 ecr 63373335], length 2856 > 20:43:27.130506 IP6 HOST> GUEST: ICMP6, packet too big, mtu 1500, length 1240 > 20:43:27.131965 IP6 DEST.22> GUEST.59864: Flags [.], ack 69357, win 501, options [nop,nop,TS val 63373335 ecr 4294920144], length 0 > 20:43:27.131996 IP6 DEST.22> GUEST.59864: Flags [.], ack 70785, win 501, options [nop,nop,TS val 63373335 ecr 4294920144], length 0 > 20:43:27.132651 IP6 GUEST.59864> DEST.22: Flags [.], seq 73641:76497, ack 2128, win 327, options [nop,nop,TS val 4294920148 ecr 63373335], length 2856 > 20:43:27.132704 IP6 HOST> GUEST: ICMP6, packet too big, mtu 1500, length 1240 > 20:43:27.346347 IP6 GUEST.59864> DEST.22: Flags [.], seq 70785:72213, ack 2128, win 327, options [nop,nop,TS val 4294920202 ecr 63373335], length 1428 > 20:43:27.360411 IP6 DEST.22> GUEST.59864: Flags [.], ack 72213, win 501, options [nop,nop,TS val 63373358 ecr 4294920202], length 0 > 20:43:27.361045 IP6 GUEST.59864> DEST.22: Flags [.], seq 76497:79353, ack 2128, win 327, options [nop,nop,TS val 4294920205 ecr 63373358], length 2856 > > So, the guest sends packets of 2856 bytes, which are too large to pass > through the eth0 of the host (1500 bytes). Thus, the host rejects the > packet and sends back an ICMPv6 packet too big, which sometimes makes > the guest send at least one packet with the small MTU, but sometimes it > doesn't. This is repeated all over and slows down the process. > > According to ethtool GSO is enabled on the guest, but disabling it using > ethtool does not have any effect. But giving the kernel > "virtio_net.gso=0" in append fixes the issue completely. > > Anyone having any ideas? > > I ran into a similar problem when using Intel e1000e driver on the host. Found a bug in most of the intel ethernet drivers when handling large GSO IPv6 packets. You can see the fix i submitted here http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8e1e8a4779cb23c1d9f51e9223795e07ec54d77a But you seem to be using Intel e100 driver. So i am not sure you are running into the same issue. You could try disabling both tso and gso on the guest using ethtool. On the host, do you have gso/tso enabled on the uplink e100 driver? Thanks Sridhar