user space virtio-net exits with "truncating packet" error

user space virtio-net exits with "truncating packet" error

[Or Gerlitz]

Hi,

When running with jumbo frames (i.e set tap0 and guest nic mtu to 9k)
and using 8k sized packets with iperf, the qemu process exits with
"virtio-net truncating packet" which I see in the code of qemu/hw/virtio-net.c
:: virtio_net_receive(). This happens only when the VM is receiving, if I
send 8K packets from the VM things go fine.

I use virtio based NIC in the VM and Linux 2.6.29.1 in both the VM and the host.
Qemu is the one provided by kvm release 84 - whose sources don't point me
to a specific git tree nor a maintainer, so I hope you can help me...

thanks,

Or.

Re: user space virtio-net exits with "truncating packet" error

[Mark McLoughlin]

On Thu, 2009-05-14 at 13:07 +0300, Or Gerlitz wrote:
> Hi,
> 
> When running with jumbo frames (i.e set tap0 and guest nic mtu to 9k)
> and using 8k sized packets with iperf, the qemu process exits with
> "virtio-net truncating packet" which I see in the code of qemu/hw/virtio-net.c
> :: virtio_net_receive(). This happens only when the VM is receiving, if I
> send 8K packets from the VM things go fine.
> 
> I use virtio based NIC in the VM and Linux 2.6.29.1 in both the VM and the host.
> Qemu is the one provided by kvm release 84

That sounds like a bug in qemu's mergeable receive buffers
implementation - the host is running out of buffers for the packet, even
though it supposedly has already checked that there is enough buffers
available on the ring.

Hmm, I think I see the bug - does the patch below work? Please try
several mtu values around the 9k mark to be sure

Cheers,
Mark.

diff --git a/hw/virtio-net.c b/hw/virtio-net.c
index f125edc..3ffd2c0 100644
--- a/hw/virtio-net.c
+++ b/hw/virtio-net.c
@@ -438,16 +438,16 @@ static void virtio_net_receive(void *opaque, const uint8_t *buf, int size)
     struct virtio_net_hdr_mrg_rxbuf *mhdr = NULL;
     size_t hdr_len, offset, i;
 
-    if (!do_virtio_net_can_receive(n, size))
+    /* hdr_len refers to the header we supply to the guest */
+    hdr_len = n->mergeable_rx_bufs ?
+        sizeof(struct virtio_net_hdr_mrg_rxbuf) : sizeof(struct virtio_net_hdr);
+
+    if (!do_virtio_net_can_receive(n, size + hdr_len))
         return;
 
     if (!receive_filter(n, buf, size))
         return;
 
-    /* hdr_len refers to the header we supply to the guest */
-    hdr_len = n->mergeable_rx_bufs ?
-        sizeof(struct virtio_net_hdr_mrg_rxbuf) : sizeof(struct virtio_net_hdr);
-
     offset = i = 0;
 
     while (offset < size) {

Re: user space virtio-net exits with "truncating packet" error

[Or Gerlitz]

Mark McLoughlin wrote:
> That sounds like a bug in qemu's mergeable receive buffers
> implementation - the host is running out of buffers for the packet, even
> though it supposedly has already checked that there is enough buffers
> available on the ring.
> 
> Hmm, I think I see the bug - does the patch below work? Please try
> several mtu values around the 9k mark to be sure

No, it doesn't help, same crash.

Or.

Re: user space virtio-net exits with "truncating packet" error

[Rusty Russell]

On Thu, 14 May 2009 07:37:42 pm Or Gerlitz wrote:
> Hi,
>
> When running with jumbo frames (i.e set tap0 and guest nic mtu to 9k)
> and using 8k sized packets with iperf, the qemu process exits with
> "virtio-net truncating packet" which I see in the code of
> qemu/hw/virtio-net.c
>
> :: virtio_net_receive(). This happens only when the VM is receiving, if I
>
> send 8K packets from the VM things go fine.
>
> I use virtio based NIC in the VM and Linux 2.6.29.1 in both the VM and the
> host. Qemu is the one provided by kvm release 84 - whose sources don't
> point me to a specific git tree nor a maintainer, so I hope you can help
> me...

Maintainer Cc'd.  The answer is that virtio_net by default only supports 1500 
MTU; I've not tried larger MTUs.

Rusty.

Re: user space virtio-net exits with "truncating packet" error

[Or Gerlitz]

Rusty Russell wrote:
> The answer is that virtio_net by default only supports 1500 
> MTU; I've not tried larger MTUs.

Rusty,

I hoped to get some performance boost from using checksum and large-send offloads 
as an alternative to jumbo frames. Looking in the virtio-net kernel driver, I see that the probe
function checks if virtio_has_feature VIRTIO_NET_F_CSUM ... VIRTIO_NET_F_HOST_TSO ... and if yes
sets the relevant bits in the NIC features mask. Looking in the virtio qemu code, I also see some offload
related code.

In my case, both the guest and the host run 2.6.29.1 and I use ethX (igb based) -- bridge -- tapY <--> qemu configuration, where qemu is the one provided by kvm latest release (84).  Now, the virtio guest NIC doesn't expose any features (its mask being 0x20 - only highdma). Anything I can do to have offloads support for my virtio environment? 

Or.

Re: user space virtio-net exits with "truncating packet" error

[Or Gerlitz]

On Thu, May 14, 2009 at 5:04 PM, Or Gerlitz <ogerlitz@voltaire.com> wrote:

> In my case, both the guest and the host run 2.6.29.1 and I use ethX (igb based) --
> bridge -- tapY <--> qemu configuration

oops, this failure happens when I used IP forwarding scheme between
tap0 to IB NIC which BTW didn't support offloads the way it was
configured. I don't see how it matters for the crash, but I will
retest with a bridge based config early next week and let you know.

Or.

Re: user space virtio-net exits with "truncating packet" error

[Rusty Russell]

On Thu, 14 May 2009 11:34:48 pm Or Gerlitz wrote:
> Rusty Russell wrote:
> > The answer is that virtio_net by default only supports 1500
> > MTU; I've not tried larger MTUs.
>
> Rusty,
>
> I hoped to get some performance boost from using checksum and large-send
> offloads as an alternative to jumbo frames. Looking in the virtio-net
> kernel driver, I see that the probe function checks if virtio_has_feature
> VIRTIO_NET_F_CSUM ... VIRTIO_NET_F_HOST_TSO ... and if yes sets the
> relevant bits in the NIC features mask. Looking in the virtio qemu code, I
> also see some offload related code.

Yes, which is why MTU is a bit misleading.

This patch may help diagnostics tho.

Cheers,
Rusty.

virtio: expose features in sysfs

Each device negotiates feature bits; expose these in sysfs to help
diagnostics and debugging.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
---
 drivers/virtio/virtio.c |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -31,11 +31,27 @@ static ssize_t modalias_show(struct devi
 	return sprintf(buf, "virtio:d%08Xv%08X\n",
 		       dev->id.device, dev->id.vendor);
 }
+static ssize_t features_show(struct device *_d,
+			     struct device_attribute *attr, char *buf)
+{
+	struct virtio_device *dev = container_of(_d, struct virtio_device, dev);
+	unsigned int i;
+	ssize_t len = 0;
+
+	/* We actually represent this as a bitstring, as it could be
+	 * arbitrary length in future. */
+	for (i = 0; i < ARRAY_SIZE(dev->features)*BITS_PER_LONG; i++)
+		len += sprintf(buf+len, "%c",
+			       test_bit(i, dev->features) ? '1' : '0');
+	len += sprintf(buf+len, "\n");
+	return len;
+}
 static struct device_attribute virtio_dev_attrs[] = {
 	__ATTR_RO(device),
 	__ATTR_RO(vendor),
 	__ATTR_RO(status),
 	__ATTR_RO(modalias),
+	__ATTR_RO(features),
 	__ATTR_NULL
 };
 

Re: user space virtio-net exits with "truncating packet" error

[Or Gerlitz]

Rusty Russell wrote:
> This patch may help diagnostics tho.
> virtio: expose features in sysfs
> 
> Each device negotiates feature bits; expose these in sysfs to help
> diagnostics and debugging.

okay, this is the info from my guest after applying the patch on its kernel.

Or.


[root@vm1 ~]# lspci | grep Ethernet
00:03.0 Ethernet controller: Qumranet, Inc. Virtio network device

[root@vm1 ~]# cat /sys/class/net/eth0/features
0x20

[root@vm1 ~]# cat /sys/class/net/eth0/device/features
0000010000000000000000001000000000000000000000000000000000000000

Re: user space virtio-net exits with "truncating packet" error

[Rusty Russell]

On Tue, 19 May 2009 09:49:02 pm Or Gerlitz wrote:
> okay, this is the info from my guest after applying the patch on its
> kernel.
...
> [root@vm1 ~]# cat /sys/class/net/eth0/device/features
> 0000010000000000000000001000000000000000000000000000000000000000

Bit 5 = VIRTIO_NET_F_MAC (the host set the mac address)
Bit 24 = VIRTIO_F_NOTIFY_ON_EMPTY

You don't have any GSO or checksum offload here.

Rusty.

Re: user space virtio-net exits with "truncating packet" error

[Or Gerlitz]

Rusty Russell wrote:
> Bit 5 = VIRTIO_NET_F_MAC (the host set the mac address)
> Bit 24 = VIRTIO_F_NOTIFY_ON_EMPTY
> You don't have any GSO or checksum offload here
Just to make sure I'm in the correct direction - I need to cause these 
offloads to be advertised by the "lower" part of virtio (e.g the qemu 
virtio code) to  the "upper" part (the quest kernel), correct? I 
understand that one of them is called front-end and the other back-end, 
but my intuitions don't go up to saying who's what...

Or.

Re: user space virtio-net exits with "truncating packet" error

[Avi Kivity]

Or Gerlitz wrote:
> Rusty Russell wrote:
>> Bit 5 = VIRTIO_NET_F_MAC (the host set the mac address)
>> Bit 24 = VIRTIO_F_NOTIFY_ON_EMPTY
>> You don't have any GSO or checksum offload here
> Just to make sure I'm in the correct direction - I need to cause these 
> offloads to be advertised by the "lower" part of virtio (e.g the qemu 
> virtio code) to  the "upper" part (the quest kernel), correct? 

Yes.  You can do that by running a recent kernel on the host, and 
compiling qemu with the headers from that kernel (you can generate them 
with make headers-install).

> I understand that one of them is called front-end and the other 
> back-end, but my intuitions don't go up to saying who's what...
>

My preferred terms are driver (in the guest) and device (in the host), 
to mimic real hardware.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: user space virtio-net exits with "truncating packet" error

[Rusty Russell]

On Wed, 20 May 2009 03:50:23 pm Or Gerlitz wrote:
> Rusty Russell wrote:
> > Bit 5 = VIRTIO_NET_F_MAC (the host set the mac address)
> > Bit 24 = VIRTIO_F_NOTIFY_ON_EMPTY
> > You don't have any GSO or checksum offload here
>
> Just to make sure I'm in the correct direction - I need to cause these
> offloads to be advertised by the "lower" part of virtio (e.g the qemu
> virtio code) to  the "upper" part (the quest kernel), correct? I
> understand that one of them is called front-end and the other back-end,
> but my intuitions don't go up to saying who's what...

Yes, that nomenclature is a bit weird.  I prefer "driver" (aka. guest, aka 
front-end) and "device" (aka. host, aka back-end).

All virtio_net drivers offer some features (at least CSUM offload), and 2.6.26 
and above will offer some serious GSO features.

Cheers,
Rusty.

Re: user space virtio-net exits with "truncating packet" error

[Michael Tokarev]

Or Gerlitz wrote:
> Rusty Russell wrote:
>> The answer is that virtio_net by default only supports 1500 
>> MTU; I've not tried larger MTUs.
> 
> Rusty,
> 
> I hoped to get some performance boost from using checksum and large-send offloads 
> as an alternative to jumbo frames. Looking in the virtio-net kernel driver, I see that the probe
> function checks if virtio_has_feature VIRTIO_NET_F_CSUM ... VIRTIO_NET_F_HOST_TSO ... and if yes
> sets the relevant bits in the NIC features mask. Looking in the virtio qemu code, I also see some offload
> related code.

A trap which I've seen before.

When compiling kvm userspace, ensure your kernel headers installed in
/usr/include has the latest if_tun.h bits, in particular the TUNSETOFFLOAD
and IFF_VNET_HDR definitions.

One solution to this is just to copy that file from kernel-2.6.29.  Or
maybe it's easier to add some -I flags to kvm compile commandline.

And one more note: I'm not sure if there's anything else needed.
Last time I was there -- it was with kvm-84 times I think, maybe
something changed since that.

/mjt

Re: user space virtio-net exits with "truncating packet" error

[Or Gerlitz]

Michael Tokarev wrote:
> A trap which I've seen before. When compiling kvm userspace, ensure your kernel headers installed in
> /usr/include has the latest if_tun.h bits, in particular the TUNSETOFFLOAD
> and IFF_VNET_HDR definitions.

OKay, this seems to be part of my problems... the kvm-85 userspace build uses the header from /usr/include
where <linux/if_tun.h> is dated back to 2001 such that its way different from the 2.6.29.1 one, such that 
TUNSETOFFLOAD and IFF_VNET_HDR are not defined. This means that the qemu code doesn't attempt to issue TUNSETOFFLOAD, and doesn't include <linux/virtio_net.h> which has plus probably more implications. 

> One solution to this is just to copy that file from kernel-2.6.29.  Or
> maybe it's easier to add some -I flags to kvm compile commandline.

This would not be enough since it needs also the virtio_net header and I'm not clear what version, the kernel one  on the one that comes with qemu... Now, all this comes into play only with the qemu provided by kvm releases since the savannah one doesn't have the code that uses TUNSETOFFLOAD, IFF_VNET_HDR etc.

Avi - do you guys have some writeup / guidelines how to make all this work?
 
Or.

Re: user space virtio-net exits with "truncating packet" error

[Avi Kivity]

Or Gerlitz wrote:
>> One solution to this is just to copy that file from kernel-2.6.29.  Or
>> maybe it's easier to add some -I flags to kvm compile commandline.
>>     
>
> This would not be enough since it needs also the virtio_net header and I'm not clear what version, the kernel one  on the one that comes with qemu... Now, all this comes into play only with the qemu provided by kvm releases since the savannah one doesn't have the code that uses TUNSETOFFLOAD, IFF_VNET_HDR etc.
>
> Avi - do you guys have some writeup / guidelines how to make all this work?
>  

Modern distros have this sewn up correctly.  For example my desktop runs 
kernel-2.6.27.21-170.2.56.fc10.x86_64, and has matching headers from 
kernel-headers-2.6.27.21-170.2.56.fc10.x86_64.

What are you using?


-- 
error compiling committee.c: too many arguments to function

Re: user space virtio-net exits with "truncating packet" error

[Sridhar Samudrala]

On Thu, 2009-05-14 at 21:28 +0930, Rusty Russell wrote:
> On Thu, 14 May 2009 07:37:42 pm Or Gerlitz wrote:
> > Hi,
> >
> > When running with jumbo frames (i.e set tap0 and guest nic mtu to 9k)
> > and using 8k sized packets with iperf, the qemu process exits with
> > "virtio-net truncating packet" which I see in the code of
> > qemu/hw/virtio-net.c
> >
> > :: virtio_net_receive(). This happens only when the VM is receiving, if I
> >
> > send 8K packets from the VM things go fine.
> >
> > I use virtio based NIC in the VM and Linux 2.6.29.1 in both the VM and the
> > host. Qemu is the one provided by kvm release 84 - whose sources don't
> > point me to a specific git tree nor a maintainer, so I hope you can help
> > me...
> 
> Maintainer Cc'd.  The answer is that virtio_net by default only supports 1500 
> MTU; I've not tried larger MTUs.


I am able to get virtio-net to work with an MTU upto 65521 between host and guest 
using qemu-kvm git tree and linux-2.6.30-rc5 running on the host and guest.
I didn't run into any issues while running iperf or netperf  with larger than 8K
message sizes.

I changed the mtu of the bridge and tap device on the host and the guest 
virtio-net device to 65521.

Thanks
Sridhar