[Qemu-devel] virtio breakage with 2.6.25 guest kernel

[Qemu-devel] virtio breakage with 2.6.25 guest kernel

[Alexander Graf]

Hi,

while I don't fully understand the problem, here's what I experience so far:

When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on tap
breaks with current KVM git, while it used to work before (haven't
bisected, definitely worked in kvm-78, but is probably due to Anthony's
rewrite). It shows the following message (comes from qemu):

virtio-net header not in first element

This is because qemu expects a message with mergeable rx bufs (12
bytes), but if I see things correctly the old kernel doesn't support
that feature (sends 10 bytes). I put in some debug messages on IO
reads/writes in the qemu virtio handler and got these:

virtio IO read: 0 = 0x100bba3
virtio IO write: 0x4 = 0x100bba3
virtio: setting features 0x100bba3

So I believe the feature masking is somewhat non-functional? Or did I
read the masking code in qemu's virtio-net wrong?

Basically for me, current KVM git broke old guest support, which is a
clear regression and should not happen. Does anybody with more expertise
in virtio want to take on this?

Alex

Re: [Qemu-devel] virtio breakage with 2.6.25 guest kernel

[Mark McLoughlin]

Hi Alex,

(Dropping kvm-devel@sf.net - it's kvm@vger.kernel.org now anyway)

On Tue, 2009-01-13 at 14:33 +0100, Alexander Graf wrote:

> while I don't fully understand the problem, here's what I experience so far:
> 
> When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on tap
> breaks with current KVM git, while it used to work before (haven't
> bisected, definitely worked in kvm-78, but is probably due to Anthony's
> rewrite). It shows the following message (comes from qemu):
> 
> virtio-net header not in first element
> 
> This is because qemu expects a message with mergeable rx bufs (12
> bytes), but if I see things correctly the old kernel doesn't support
> that feature (sends 10 bytes). I put in some debug messages on IO
> reads/writes in the qemu virtio handler and got these:
> 
> virtio IO read: 0 = 0x100bba3
> virtio IO write: 0x4 = 0x100bba3
> virtio: setting features 0x100bba3
> 
> So I believe the feature masking is somewhat non-functional? Or did I
> read the masking code in qemu's virtio-net wrong?
> 
> Basically for me, current KVM git broke old guest support, which is a
> clear regression and should not happen. Does anybody with more expertise
> in virtio want to take on this?

Yeah, I noticed this one last week.

The issue here is that 2.6.25 virtio erroneously says it supports all
features supported by the host - i.e. it's a guest kernel bug.

With recent qemu, that causes us to enable mergeable receive buffers and
the guest barfs. See:

        mask = ioread32(vp_dev->ioaddr + VIRTIO_PCI_HOST_FEATURES);
        if (mask & (1 << bit)) {
                mask |= (1 << bit);
                iowrite32(mask, vp_dev->ioaddr + VIRTIO_PCI_GUEST_FEATURES);

This got fixed in 2.6.26 by:

  http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c45a6816

I did consider forwarding this to stable@vger.kernel.org, but apparently
the 2.6.25.x series is finished now anyway.

Cheers,
Mark.

Re: [Qemu-devel] virtio breakage with 2.6.25 guest kernel

[Alexander Graf]

Mark McLoughlin wrote:
> Hi Alex,
>
> (Dropping kvm-devel@sf.net - it's kvm@vger.kernel.org now anyway)
>   

Thunderbird keeps selecting that one and I'm sometimes too confused to
see it's actually the wrong one.

> On Tue, 2009-01-13 at 14:33 +0100, Alexander Graf wrote:
>
>   
>> while I don't fully understand the problem, here's what I experience so far:
>>
>> When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on tap
>> breaks with current KVM git, while it used to work before (haven't
>> bisected, definitely worked in kvm-78, but is probably due to Anthony's
>> rewrite). It shows the following message (comes from qemu):
>>
>> virtio-net header not in first element
>>
>> This is because qemu expects a message with mergeable rx bufs (12
>> bytes), but if I see things correctly the old kernel doesn't support
>> that feature (sends 10 bytes). I put in some debug messages on IO
>> reads/writes in the qemu virtio handler and got these:
>>
>> virtio IO read: 0 = 0x100bba3
>> virtio IO write: 0x4 = 0x100bba3
>> virtio: setting features 0x100bba3
>>
>> So I believe the feature masking is somewhat non-functional? Or did I
>> read the masking code in qemu's virtio-net wrong?
>>
>> Basically for me, current KVM git broke old guest support, which is a
>> clear regression and should not happen. Does anybody with more expertise
>> in virtio want to take on this?
>>     
>
> Yeah, I noticed this one last week.
>
> The issue here is that 2.6.25 virtio erroneously says it supports all
> features supported by the host - i.e. it's a guest kernel bug.
>
> With recent qemu, that causes us to enable mergeable receive buffers and
> the guest barfs. See:
>
>         mask = ioread32(vp_dev->ioaddr + VIRTIO_PCI_HOST_FEATURES);
>         if (mask & (1 << bit)) {
>                 mask |= (1 << bit);
>                 iowrite32(mask, vp_dev->ioaddr + VIRTIO_PCI_GUEST_FEATURES);
>
> This got fixed in 2.6.26 by:
>
>   http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=c45a6816
>
> I did consider forwarding this to stable@vger.kernel.org, but apparently
> the 2.6.25.x series is finished now anyway.
>   

Ok, that's good to know. I don't think anyone officially supported
virtio in 2.6.25 days, but I'll check internally if we need to backport
anything.

Thanks a lot

Alex

Re: [Qemu-devel] virtio breakage with 2.6.25 guest kernel

[Avi Kivity]

Alexander Graf wrote:
> Hi,
>
> while I don't fully understand the problem, here's what I experience so far:
>
> When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on tap
> breaks with current KVM git, while it used to work before (haven't
> bisected, definitely worked in kvm-78, but is probably due to Anthony's
> rewrite). It shows the following message (comes from qemu):
>
> virtio-net header not in first element
>
> This is because qemu expects a message with mergeable rx bufs (12
> bytes), but if I see things correctly the old kernel doesn't support
> that feature (sends 10 bytes). I put in some debug messages on IO
> reads/writes in the qemu virtio handler and got these:
>
> virtio IO read: 0 = 0x100bba3
> virtio IO write: 0x4 = 0x100bba3
> virtio: setting features 0x100bba3
>
> So I believe the feature masking is somewhat non-functional? Or did I
> read the masking code in qemu's virtio-net wrong?
>
> Basically for me, current KVM git broke old guest support, which is a
> clear regression and should not happen. Does anybody with more expertise
> in virtio want to take on this?
>   

I think this was reported and fixed, possibly by c45a681.

-- 
error compiling committee.c: too many arguments to function

[Qemu-devel] Re: virtio breakage with 2.6.25 guest kernel

[Anthony Liguori]

Alexander Graf wrote:
> Hi,
>
> while I don't fully understand the problem, here's what I experience so far:
>
> When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on tap
> breaks with current KVM git, while it used to work before (haven't
> bisected, definitely worked in kvm-78, but is probably due to Anthony's
> rewrite). It shows the following message (comes from qemu):
>   

There were a couple of old-guest-breaking regressions.  I think we've 
fixed all of them but there could be more.  Are you using the latest 
kvm-userspace?

Regards,

Anthony Liguori

> virtio-net header not in first element
>
> This is because qemu expects a message with mergeable rx bufs (12
> bytes), but if I see things correctly the old kernel doesn't support
> that feature (sends 10 bytes). I put in some debug messages on IO
> reads/writes in the qemu virtio handler and got these:
>
> virtio IO read: 0 = 0x100bba3
> virtio IO write: 0x4 = 0x100bba3
> virtio: setting features 0x100bba3
>
> So I believe the feature masking is somewhat non-functional? Or did I
> read the masking code in qemu's virtio-net wrong?
>
> Basically for me, current KVM git broke old guest support, which is a
> clear regression and should not happen. Does anybody with more expertise
> in virtio want to take on this?
>
> Alex
>   

[Qemu-devel] Re: virtio breakage with 2.6.25 guest kernel

[Alexander Graf]

On 13.01.2009, at 21:14, Anthony Liguori <anthony@codemonkey.ws> wrote:

> Alexander Graf wrote:
>> Hi,
>>
>> while I don't fully understand the problem, here's what I  
>> experience so far:
>>
>> When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on  
>> tap
>> breaks with current KVM git, while it used to work before (haven't
>> bisected, definitely worked in kvm-78, but is probably due to  
>> Anthony's
>> rewrite). It shows the following message (comes from qemu):
>>
>
> There were a couple of old-guest-breaking regressions.  I think  
> we've fixed all of them but there could be more.  Are you using the  
> latest kvm-userspace?

This one is definitely due to the broken guest kernel. I applied the  
patch mark mentioned to ours and things started working.

So the only way I can think of to 'fix' it is by detecting broken  
guests. We could supply a host mask of 0xffffffff and see if tge guest  
feature mask is tge same. If so, feature masking is probably broken.

Alex

>
>
> Regards,
>
> Anthony Liguori
>
>> virtio-net header not in first element
>>
>> This is because qemu expects a message with mergeable rx bufs (12
>> bytes), but if I see things correctly the old kernel doesn't support
>> that feature (sends 10 bytes). I put in some debug messages on IO
>> reads/writes in the qemu virtio handler and got these:
>>
>> virtio IO read: 0 = 0x100bba3
>> virtio IO write: 0x4 = 0x100bba3
>> virtio: setting features 0x100bba3
>>
>> So I believe the feature masking is somewhat non-functional? Or did I
>> read the masking code in qemu's virtio-net wrong?
>>
>> Basically for me, current KVM git broke old guest support, which is a
>> clear regression and should not happen. Does anybody with more  
>> expertise
>> in virtio want to take on this?
>>
>> Alex
>>
>

[Qemu-devel] Re: virtio breakage with 2.6.25 guest kernel

[Mark McLoughlin]

On Tue, 2009-01-13 at 22:05 +0100, Alexander Graf wrote:

> On 13.01.2009, at 21:14, Anthony Liguori <anthony@codemonkey.ws> wrote:
> 
> > Alexander Graf wrote:
> >> Hi,
> >>
> >> while I don't fully understand the problem, here's what I  
> >> experience so far:
> >>
> >> When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on  
> >> tap
> >> breaks with current KVM git, while it used to work before (haven't
> >> bisected, definitely worked in kvm-78, but is probably due to  
> >> Anthony's
> >> rewrite). It shows the following message (comes from qemu):
> >>
> >
> > There were a couple of old-guest-breaking regressions.  I think  
> > we've fixed all of them but there could be more.  Are you using the  
> > latest kvm-userspace?

See:

  http://lists.gnu.org/archive/html/qemu-devel/2009-01/msg00574.html

> This one is definitely due to the broken guest kernel. I applied the  
> patch mark mentioned to ours and things started working.
> 
> So the only way I can think of to 'fix' it is by detecting broken  
> guests. We could supply a host mask of 0xffffffff and see if tge guest  
> feature mask is tge same. If so, feature masking is probably broken.

Nice idea, but no way of making the guest through feature detection
negotiation, I don't think.

Cheers,
Mark.

[Qemu-devel] Re: virtio breakage with 2.6.25 guest kernel

[Rusty Russell]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

On Wednesday 14 January 2009 19:52:58 Mark McLoughlin wrote:
> On Tue, 2009-01-13 at 22:05 +0100, Alexander Graf wrote:
> 
> > On 13.01.2009, at 21:14, Anthony Liguori <anthony@codemonkey.ws> wrote:
> > 
> > > Alexander Graf wrote:
> > >> Hi,
> > >>
> > >> while I don't fully understand the problem, here's what I  
> > >> experience so far:
> > >>
> > >> When using an openSUSE 11.0 kernel (2.6.25) in the guest, virtio on  
> > >> tap
> > >> breaks with current KVM git, while it used to work before (haven't
> > >> bisected, definitely worked in kvm-78, but is probably due to  
> > >> Anthony's
> > >> rewrite). It shows the following message (comes from qemu):
> > >>
> > >
> > > There were a couple of old-guest-breaking regressions.  I think  
> > > we've fixed all of them but there could be more.  Are you using the  
> > > latest kvm-userspace?
> 
> See:
> 
>   http://lists.gnu.org/archive/html/qemu-devel/2009-01/msg00574.html
> 
> > This one is definitely due to the broken guest kernel. I applied the  
> > patch mark mentioned to ours and things started working.
> > 
> > So the only way I can think of to 'fix' it is by detecting broken  
> > guests. We could supply a host mask of 0xffffffff and see if tge guest  
> > feature mask is tge same. If so, feature masking is probably broken.
> 
> Nice idea, but no way of making the guest through feature detection
> negotiation, I don't think.

Just add a feature "VIRTIO_F_LIES_ABOUT_FEATURES"?  No guest should ever set this.

But I'm not sure it's worth the pain...
Rusty.

[-- Attachment #2: Type: text/html, Size: 7972 bytes --]