TAP MTU >= 4055 problem?

TAP MTU >= 4055 problem?

[Matthew Faulkner]

Hey all,
I'm using KVM to setup a virtual machine. Using this machine i'm
attempting to setup a connection to host machine using the tap
interface. I setup the interface using the standard tap method and
then start the KVM using the following:

kvm -net nic,model=e1000 -net tap,ifname=qtap0,script/etc/kvm/kvm-ifup
-drive file=./image.img

When the virual machine starts i setup the IP addresses (host =
192.168.100.1 vm = 192.168.100.2) and I am able to bring between the
host and virtual machine. However, what i need to do is to be able to
send jumbo frames between the two. I therefore increase the MTU size
on both the host machine and virutal machine to 9000.

Now when i try to do from within the host

ping -s 4096 192.168.100.1


I go no respone. So i started with a lower packet size and figured out
below a size of 4054 packets were sent and recevied (without ip
fragmentation), however, as soon as the packets were >= 4055 it
stopped working.

Is this a known problem? Have I set something up incorrectly?

Thanks for your help

P.S. I'm not on the kvm mailing list so please cc me with any replies

Re: TAP MTU >= 4055 problem?

[Javier Guerra]

On Thu, Oct 30, 2008 at 4:41 AM, Matthew Faulkner
<matthew.faulkner@gmail.com> wrote:
> I go no respone. So i started with a lower packet size and figured out
> below a size of 4054 packets were sent and recevied (without ip
> fragmentation), however, as soon as the packets were >= 4055 it
> stopped working.
>
> Is this a known problem? Have I set something up incorrectly?

are you using a bridge?  it has some problems with long packets.

if you can, try without a bridge.  if not, make sure that you set the
MTU of both ethX and tapX to 9000 before adding anything to the
bridge.  i think the bridge itself has its own MTU that is set
according to the interfaces it gets; but i don't know if it can change
MTU when an interface changes.


-- 
Javier

Re: TAP MTU >= 4055 problem?

[Matthew Faulkner]

I'm not using a bridge. And i have set the mtu on both tapX and ethX to 9000.

Having done a little packet sniffing from the hostmachine -> virtual
machine I can see that packets of the correct (large) size are being
sent. Whilst sniffing on the virtual machine I don't see any packets.
This indicates to me that the network emulation drivers in KVM aren't
setup correctly?

I'm using the e1000 drivers.

Anyone any thoughts?

2008/10/30 Javier Guerra <javier@guerrag.com>:
> On Thu, Oct 30, 2008 at 4:41 AM, Matthew Faulkner
> <matthew.faulkner@gmail.com> wrote:
>> I go no respone. So i started with a lower packet size and figured out
>> below a size of 4054 packets were sent and recevied (without ip
>> fragmentation), however, as soon as the packets were >= 4055 it
>> stopped working.
>>
>> Is this a known problem? Have I set something up incorrectly?
>
> are you using a bridge?  it has some problems with long packets.
>
> if you can, try without a bridge.  if not, make sure that you set the
> MTU of both ethX and tapX to 9000 before adding anything to the
> bridge.  i think the bridge itself has its own MTU that is set
> according to the interfaces it gets; but i don't know if it can change
> MTU when an interface changes.
>
>
> --
> Javier
>

Re: TAP MTU >= 4055 problem?

[Matthew Faulkner]

For additionial information, i've placed some printf's within KVMs
e1000 emualted drivers.

It turns out the within the e1000 receive function (e1000_receive) the
correct size is received. Somewhere between there and the upper layers
something goes wrong. I am continuing to investiage this problem, but
if anyone else shed some light on this I would be greatful.


2008/10/30 Matthew Faulkner <matthew.faulkner@gmail.com>:
> I'm not using a bridge. And i have set the mtu on both tapX and ethX to 9000.
>
> Having done a little packet sniffing from the hostmachine -> virtual
> machine I can see that packets of the correct (large) size are being
> sent. Whilst sniffing on the virtual machine I don't see any packets.
> This indicates to me that the network emulation drivers in KVM aren't
> setup correctly?
>
> I'm using the e1000 drivers.
>
> Anyone any thoughts?
>
> 2008/10/30 Javier Guerra <javier@guerrag.com>:
>> On Thu, Oct 30, 2008 at 4:41 AM, Matthew Faulkner
>> <matthew.faulkner@gmail.com> wrote:
>>> I go no respone. So i started with a lower packet size and figured out
>>> below a size of 4054 packets were sent and recevied (without ip
>>> fragmentation), however, as soon as the packets were >= 4055 it
>>> stopped working.
>>>
>>> Is this a known problem? Have I set something up incorrectly?
>>
>> are you using a bridge?  it has some problems with long packets.
>>
>> if you can, try without a bridge.  if not, make sure that you set the
>> MTU of both ethX and tapX to 9000 before adding anything to the
>> bridge.  i think the bridge itself has its own MTU that is set
>> according to the interfaces it gets; but i don't know if it can change
>> MTU when an interface changes.
>>
>>
>> --
>> Javier
>>
>

Re: TAP MTU >= 4055 problem?

[Matthew Faulkner]

Hey all,

I've solved the problem. Turns out the tap buffers were set at the
size 4096. When pulling a packet off that is greater than 4096 the 2nd
packet (for some reason) is thrown aaway or ignored.

I've done a hacky solutoin by simply increasing this number of 65536.



2008/10/30 Matthew Faulkner <matthew.faulkner@gmail.com>:
> For additionial information, i've placed some printf's within KVMs
> e1000 emualted drivers.
>
> It turns out the within the e1000 receive function (e1000_receive) the
> correct size is received. Somewhere between there and the upper layers
> something goes wrong. I am continuing to investiage this problem, but
> if anyone else shed some light on this I would be greatful.
>
>
> 2008/10/30 Matthew Faulkner <matthew.faulkner@gmail.com>:
>> I'm not using a bridge. And i have set the mtu on both tapX and ethX to 9000.
>>
>> Having done a little packet sniffing from the hostmachine -> virtual
>> machine I can see that packets of the correct (large) size are being
>> sent. Whilst sniffing on the virtual machine I don't see any packets.
>> This indicates to me that the network emulation drivers in KVM aren't
>> setup correctly?
>>
>> I'm using the e1000 drivers.
>>
>> Anyone any thoughts?
>>
>> 2008/10/30 Javier Guerra <javier@guerrag.com>:
>>> On Thu, Oct 30, 2008 at 4:41 AM, Matthew Faulkner
>>> <matthew.faulkner@gmail.com> wrote:
>>>> I go no respone. So i started with a lower packet size and figured out
>>>> below a size of 4054 packets were sent and recevied (without ip
>>>> fragmentation), however, as soon as the packets were >= 4055 it
>>>> stopped working.
>>>>
>>>> Is this a known problem? Have I set something up incorrectly?
>>>
>>> are you using a bridge?  it has some problems with long packets.
>>>
>>> if you can, try without a bridge.  if not, make sure that you set the
>>> MTU of both ethX and tapX to 9000 before adding anything to the
>>> bridge.  i think the bridge itself has its own MTU that is set
>>> according to the interfaces it gets; but i don't know if it can change
>>> MTU when an interface changes.
>>>
>>>
>>> --
>>> Javier
>>>
>>
>

[PATCH] kvm: qemu: increase the size of the tap buffer [was Re: TAP MTU >= 4055 problem?]

[Mark McLoughlin]

Hi Matthew,

On Thu, 2008-10-30 at 19:43 +0000, Matthew Faulkner wrote:
> Hey all,
> 
> I've solved the problem. Turns out the tap buffers were set at the
> size 4096. When pulling a packet off that is greater than 4096 the 2nd
> packet (for some reason) is thrown aaway or ignored.
> 
> I've done a hacky solutoin by simply increasing this number of 65536.

Thanks for debugging - that makes a whole lot of sense.

To explain in more detail - if you set the tap MTU size to 9000 and do
ping -s 4055

  1) the host's networking stack will construct an ethernet frame of
     4097 (4055 bytes ICMP data, 8 bytes ICMP header, 20 bytes TCP
     header and 14 bytes ethernet header)

  2) qemu will read the frame from the tap device into it's 4096 byte 
     buffer causing the kernel to truncate it to 4096 bytes

  3) qemu passes that on to the guest and the guest's IP stack will see 
     that there is less data in the buffer than is specified in the IP 
     header. It then silently drops the packet (see InTruncatedPkts 
     in /proc/net/netstat)

It sucks that the kernel just silently truncates and doesn't give
userspace a chance to retry with a larger buffer. We could add that as
an extension, I guess.

Best short term thing to do is just increase the tap buffer size. Patch
below. A similar patch should go into qemu upstream.

Cheers,
Mark.

From: Mark McLoughlin <markmc@redhat.com>
Subject: [PATCH] kvm: qemu: increase size of tun/tap buffer

As debugged by Matthew Faulkner, with a 4k byte tap buffer if you
increase the MTU on the tap device to greater than 4k, the packets
read by qemu into the tap buffer will be truncated and the guest
will discard the packet.

With GSO enabled, we use a 64k tap buffer, so let's just use a 64k
buffer in all cases.

Also, remove the obtuse logic for figuring out the max GSO buffer
size. We shouldn't receive IP packets larger than 64k, so let's just
use 17 pages to make sure we've enough room for headers.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
---
 qemu/vl.c |   14 ++++++--------
 1 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/qemu/vl.c b/qemu/vl.c
index da92785..4dccb21 100644
--- a/qemu/vl.c
+++ b/qemu/vl.c
@@ -4413,15 +4413,13 @@ void tap_using_vnet_hdr(void *opaque, int using_vnet_hdr)
 
 #else /* !defined(_WIN32) */
 
-#ifndef IFF_VNET_HDR
-#define TAP_BUFSIZE 4096
-#else
+/* Maximum GSO packet size (64k) plus plenty of room for
+ * the ethernet and virtio_net headers
+ */
+#define TAP_BUFSIZE (4096 + 65536)
+
+#ifdef IFF_VNET_HDR
 #include <linux/virtio_net.h>
-#define ETH_HLEN 14
-#define ETH_DATA_LEN 1500
-#define MAX_PACKET_LEN (ETH_HLEN + ETH_DATA_LEN)
-#define MAX_SKB_FRAGS ((65536/TARGET_PAGE_SIZE) + 2)
-#define TAP_BUFSIZE (sizeof(struct virtio_net_hdr) + MAX_PACKET_LEN + (MAX_SKB_FRAGS*TARGET_PAGE_SIZE))
 #endif
 
 typedef struct TAPState {
-- 
1.6.0.1

Re: [PATCH] kvm: qemu: increase the size of the tap buffer [was Re: TAP MTU >= 4055 problem?]

[Avi Kivity]

Mark McLoughlin wrote:
> It sucks that the kernel just silently truncates and doesn't give
> userspace a chance to retry with a larger buffer. We could add that as
> an extension, I guess.
>
>   

What we want even more is to give the kernel a set of buffers (iovec) 
and have it fill them as much as it can with a number of packets, giving 
us some info as to where frames begin.  Manage the whole thing using aio 
reads.

> As debugged by Matthew Faulkner, with a 4k byte tap buffer if you
> increase the MTU on the tap device to greater than 4k, the packets
> read by qemu into the tap buffer will be truncated and the guest
> will discard the packet.
>
> With GSO enabled, we use a 64k tap buffer, so let's just use a 64k
> buffer in all cases.
>
> Also, remove the obtuse logic for figuring out the max GSO buffer
> size. We shouldn't receive IP packets larger than 64k, so let's just
> use 17 pages to make sure we've enough room for headers.
>
>   

Applied, thanks.

-- 
error compiling committee.c: too many arguments to function