[Qemu-devel] Headsup: windows virtio networking does not work on current git

[Qemu-devel] Headsup: windows virtio networking does not work on current git

[Michael Tokarev]

Just a heads-up for now, no real diagnostics or anything like that.

Current git master (a9c87c586ba9ee290792a98dc126b2861b7f8b03), when booted
a windows guest, results in no virtio-net inside.  Neither winXP nor Win7,
neither older nor latest (22 Jan 2013) virtio-net drivers works.

Windows displays a yellow exclamation mark near the virtio-net device and
says it can't start the device (Code 10).

Linux guests work fine, quick test anyway.

Cc'ing Jason since his virtio-net changes was last.  But I repeat: no
diagnostics as of yet, no bisection.

Just a heads-up.

Thanks,

/mjt

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Michael Tokarev]

02.02.2013 00:18, Michael Tokarev пишет:
> Just a heads-up for now, no real diagnostics or anything like that.
>
> Current git master (a9c87c586ba9ee290792a98dc126b2861b7f8b03), when booted
> a windows guest, results in no virtio-net inside.  Neither winXP nor Win7,
> neither older nor latest (22 Jan 2013) virtio-net drivers works.
>
> Windows displays a yellow exclamation mark near the virtio-net device and
> says it can't start the device (Code 10).
>
> Linux guests work fine, quick test anyway.
>
> Cc'ing Jason since his virtio-net changes was last.  But I repeat: no
> diagnostics as of yet, no bisection.

Bisection was easy, since win works fine right before the multiqueue
virtio-net series.  This is the first bad commit:

commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
Author: Jason Wang <jasowang@redhat.com>
Date:   Wed Jan 30 19:12:39 2013 +0800

     virtio-net: multiqueue support

     This patch implements both userspace and vhost support for multiple queue
     virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
     VirtIONetQueue to VirtIONet.

     Signed-off-by: Jason Wang <jasowang@redhat.com>
     Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

After this commit, win guest (winXP and win7) shows yellow
exclamation sign and is unable to start the device with
code 10.

FWIW.  I'm not sure it is a good idea to make a release with
such a breakage, even rc0.

Thanks,

/mjt

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Anthony Liguori]

Michael Tokarev <mjt@tls.msk.ru> writes:

> 02.02.2013 00:18, Michael Tokarev пишет:
>> Just a heads-up for now, no real diagnostics or anything like that.
>>
>> Current git master (a9c87c586ba9ee290792a98dc126b2861b7f8b03), when booted
>> a windows guest, results in no virtio-net inside.  Neither winXP nor Win7,
>> neither older nor latest (22 Jan 2013) virtio-net drivers works.
>>
>> Windows displays a yellow exclamation mark near the virtio-net device and
>> says it can't start the device (Code 10).
>>
>> Linux guests work fine, quick test anyway.
>>
>> Cc'ing Jason since his virtio-net changes was last.  But I repeat: no
>> diagnostics as of yet, no bisection.
>
> Bisection was easy, since win works fine right before the multiqueue
> virtio-net series.  This is the first bad commit:

Adding Vadim and Michael.

If you use -M pc-1.3 or explicitly disable multiqueue, does the driver work?

Regards,

Anthony Liguori

>
> commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
> Author: Jason Wang <jasowang@redhat.com>
> Date:   Wed Jan 30 19:12:39 2013 +0800
>
>      virtio-net: multiqueue support
>
>      This patch implements both userspace and vhost support for multiple queue
>      virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
>      VirtIONetQueue to VirtIONet.
>
>      Signed-off-by: Jason Wang <jasowang@redhat.com>
>      Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
>
> After this commit, win guest (winXP and win7) shows yellow
> exclamation sign and is unable to start the device with
> code 10.
>
> FWIW.  I'm not sure it is a good idea to make a release with
> such a breakage, even rc0.
>
> Thanks,
>
> /mjt

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Michael Tokarev]

02.02.2013 00:36, Anthony Liguori wrote:
> Michael Tokarev <mjt@tls.msk.ru> writes:
>
>> 02.02.2013 00:18, Michael Tokarev wrote:
>>> Just a heads-up for now, no real diagnostics or anything like that.
>>>
>>> Current git master (a9c87c586ba9ee290792a98dc126b2861b7f8b03), when booted
>>> a windows guest, results in no virtio-net inside.  Neither winXP nor Win7,
>>> neither older nor latest (22 Jan 2013) virtio-net drivers works.
>>>
>>> Windows displays a yellow exclamation mark near the virtio-net device and
>>> says it can't start the device (Code 10).
>>>
>>> Linux guests work fine, quick test anyway.
>>>
>>> Cc'ing Jason since his virtio-net changes was last.  But I repeat: no
>>> diagnostics as of yet, no bisection.
>>
>> Bisection was easy, since win works fine right before the multiqueue
>> virtio-net series.  This is the first bad commit:
>
> Adding Vadim and Michael.
>
> If you use -M pc-1.3 or explicitly disable multiqueue, does the driver work?

Neither one of these nor both makes any visible difference.
Neither does -M pc-1.1 (just in case).

> Regards,
>
> Anthony Liguori
>
>>
>> commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
>> Author: Jason Wang <jasowang@redhat.com>
>> Date:   Wed Jan 30 19:12:39 2013 +0800
>>
>>       virtio-net: multiqueue support
>>
>>       This patch implements both userspace and vhost support for multiple queue
>>       virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
>>       VirtIONetQueue to VirtIONet.
>>
>>       Signed-off-by: Jason Wang <jasowang@redhat.com>
>>       Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
>>
>> After this commit, win guest (winXP and win7) shows yellow
>> exclamation sign and is unable to start the device with
>> code 10.
>>
>> FWIW.  I'm not sure it is a good idea to make a release with
>> such a breakage, even rc0.
>>
>> Thanks,
>>
>> /mjt

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Anthony Liguori]

Michael Tokarev <mjt@tls.msk.ru> writes:

> 02.02.2013 00:36, Anthony Liguori wrote:
>> Michael Tokarev <mjt@tls.msk.ru> writes:
>>
>>> 02.02.2013 00:18, Michael Tokarev wrote:
>>>> Just a heads-up for now, no real diagnostics or anything like that.
>>>>
>>>> Current git master (a9c87c586ba9ee290792a98dc126b2861b7f8b03), when booted
>>>> a windows guest, results in no virtio-net inside.  Neither winXP nor Win7,
>>>> neither older nor latest (22 Jan 2013) virtio-net drivers works.
>>>>
>>>> Windows displays a yellow exclamation mark near the virtio-net device and
>>>> says it can't start the device (Code 10).
>>>>
>>>> Linux guests work fine, quick test anyway.
>>>>
>>>> Cc'ing Jason since his virtio-net changes was last.  But I repeat: no
>>>> diagnostics as of yet, no bisection.
>>>
>>> Bisection was easy, since win works fine right before the multiqueue
>>> virtio-net series.  This is the first bad commit:
>>
>> Adding Vadim and Michael.
>>
>> If you use -M pc-1.3 or explicitly disable multiqueue, does the driver work?
>
> Neither one of these nor both makes any visible difference.
> Neither does -M pc-1.1 (just in case).

Hrm, then it's very likely not a driver problem.  Thanks.

Regards,

Anthony Liguori

>
>> Regards,
>>
>> Anthony Liguori
>>
>>>
>>> commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
>>> Author: Jason Wang <jasowang@redhat.com>
>>> Date:   Wed Jan 30 19:12:39 2013 +0800
>>>
>>>       virtio-net: multiqueue support
>>>
>>>       This patch implements both userspace and vhost support for multiple queue
>>>       virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
>>>       VirtIONetQueue to VirtIONet.
>>>
>>>       Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>       Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
>>>
>>> After this commit, win guest (winXP and win7) shows yellow
>>> exclamation sign and is unable to start the device with
>>> code 10.
>>>
>>> FWIW.  I'm not sure it is a good idea to make a release with
>>> such a breakage, even rc0.
>>>
>>> Thanks,
>>>
>>> /mjt

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Jason Wang]

On 02/02/2013 05:13 AM, Anthony Liguori wrote:
> Michael Tokarev <mjt@tls.msk.ru> writes:
>
>> 02.02.2013 00:36, Anthony Liguori wrote:
>>> Michael Tokarev <mjt@tls.msk.ru> writes:
>>>
>>>> 02.02.2013 00:18, Michael Tokarev wrote:
>>>>> Just a heads-up for now, no real diagnostics or anything like that.
>>>>>
>>>>> Current git master (a9c87c586ba9ee290792a98dc126b2861b7f8b03), when booted
>>>>> a windows guest, results in no virtio-net inside.  Neither winXP nor Win7,
>>>>> neither older nor latest (22 Jan 2013) virtio-net drivers works.
>>>>>
>>>>> Windows displays a yellow exclamation mark near the virtio-net device and
>>>>> says it can't start the device (Code 10).
>>>>>
>>>>> Linux guests work fine, quick test anyway.
>>>>>
>>>>> Cc'ing Jason since his virtio-net changes was last.  But I repeat: no
>>>>> diagnostics as of yet, no bisection.
>>>> Bisection was easy, since win works fine right before the multiqueue
>>>> virtio-net series.  This is the first bad commit:
>>> Adding Vadim and Michael.
>>>
>>> If you use -M pc-1.3 or explicitly disable multiqueue, does the driver work?
>> Neither one of these nor both makes any visible difference.
>> Neither does -M pc-1.1 (just in case).
> Hrm, then it's very likely not a driver problem.  Thanks.
>
> Regards,
>
> Anthony Liguori

Have a look at this issue. It was caused by multiqueue patch who adds a
new field to virtio_net_cfg. Not sure multiqueue is the root cause since
I also find even w/o multiqueue, adding any new field to virtio_net_cfg
will break windows guest. Haven't had a clue on this, will continue
investigate.
>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>>> commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
>>>> Author: Jason Wang <jasowang@redhat.com>
>>>> Date:   Wed Jan 30 19:12:39 2013 +0800
>>>>
>>>>       virtio-net: multiqueue support
>>>>
>>>>       This patch implements both userspace and vhost support for multiple queue
>>>>       virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
>>>>       VirtIONetQueue to VirtIONet.
>>>>
>>>>       Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>       Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
>>>>
>>>> After this commit, win guest (winXP and win7) shows yellow
>>>> exclamation sign and is unable to start the device with
>>>> code 10.
>>>>
>>>> FWIW.  I'm not sure it is a good idea to make a release with
>>>> such a breakage, even rc0.
>>>>
>>>> Thanks,
>>>>
>>>> /mjt
>

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Vadim Rozenfeld]

On Sat, 2013-02-02 at 20:42 +0800, Jason Wang wrote:
> On 02/02/2013 05:13 AM, Anthony Liguori wrote:
> > Michael Tokarev <mjt@tls.msk.ru> writes:
> >
> >> 02.02.2013 00:36, Anthony Liguori wrote:
> >>> Michael Tokarev <mjt@tls.msk.ru> writes:
> >>>
> >>>> 02.02.2013 00:18, Michael Tokarev wrote:
> >>>>> Just a heads-up for now, no real diagnostics or anything like that.
> >>>>>
> >>>>> Current git master (a9c87c586ba9ee290792a98dc126b2861b7f8b03), when booted
> >>>>> a windows guest, results in no virtio-net inside.  Neither winXP nor Win7,
> >>>>> neither older nor latest (22 Jan 2013) virtio-net drivers works.
> >>>>>
> >>>>> Windows displays a yellow exclamation mark near the virtio-net device and
> >>>>> says it can't start the device (Code 10).
> >>>>>
> >>>>> Linux guests work fine, quick test anyway.
> >>>>>
> >>>>> Cc'ing Jason since his virtio-net changes was last.  But I repeat: no
> >>>>> diagnostics as of yet, no bisection.
> >>>> Bisection was easy, since win works fine right before the multiqueue
> >>>> virtio-net series.  This is the first bad commit:
> >>> Adding Vadim and Michael.
> >>>
> >>> If you use -M pc-1.3 or explicitly disable multiqueue, does the driver work?
> >> Neither one of these nor both makes any visible difference.
> >> Neither does -M pc-1.1 (just in case).
> > Hrm, then it's very likely not a driver problem.  Thanks.
> >
> > Regards,
> >
> > Anthony Liguori
> 
> Have a look at this issue. It was caused by multiqueue patch who adds a
> new field to virtio_net_cfg. Not sure multiqueue is the root cause since
> I also find even w/o multiqueue, adding any new field to virtio_net_cfg
> will break windows guest. Haven't had a clue on this, will continue
> investigate.

cc'ing Yan, our NDIS guy.
Thank you,
Vadim.
> >
> >>> Regards,
> >>>
> >>> Anthony Liguori
> >>>
> >>>> commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
> >>>> Author: Jason Wang <jasowang@redhat.com>
> >>>> Date:   Wed Jan 30 19:12:39 2013 +0800
> >>>>
> >>>>       virtio-net: multiqueue support
> >>>>
> >>>>       This patch implements both userspace and vhost support for multiple queue
> >>>>       virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
> >>>>       VirtIONetQueue to VirtIONet.
> >>>>
> >>>>       Signed-off-by: Jason Wang <jasowang@redhat.com>
> >>>>       Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
> >>>>
> >>>> After this commit, win guest (winXP and win7) shows yellow
> >>>> exclamation sign and is unable to start the device with
> >>>> code 10.
> >>>>
> >>>> FWIW.  I'm not sure it is a good idea to make a release with
> >>>> such a breakage, even rc0.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> /mjt
> >
> 

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Anthony Liguori]

Vadim Rozenfeld <vrozenfe@redhat.com> writes:

> On Sat, 2013-02-02 at 20:42 +0800, Jason Wang wrote:
>> 
>> Have a look at this issue. It was caused by multiqueue patch who adds a
>> new field to virtio_net_cfg. Not sure multiqueue is the root cause since
>> I also find even w/o multiqueue, adding any new field to virtio_net_cfg
>> will break windows guest. Haven't had a clue on this, will continue
>> investigate.
>
> cc'ing Yan, our NDIS guy.

If it helps, mq changes the config size from 8 to 16 bytes.  If the
driver was making an assumption about an 8-byte config size, that's
likely what the problem is.

Regards,

Anthony Liguori

> Thank you,
> Vadim.
>> >
>> >>> Regards,
>> >>>
>> >>> Anthony Liguori
>> >>>
>> >>>> commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
>> >>>> Author: Jason Wang <jasowang@redhat.com>
>> >>>> Date:   Wed Jan 30 19:12:39 2013 +0800
>> >>>>
>> >>>>       virtio-net: multiqueue support
>> >>>>
>> >>>>       This patch implements both userspace and vhost support for multiple queue
>> >>>>       virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
>> >>>>       VirtIONetQueue to VirtIONet.
>> >>>>
>> >>>>       Signed-off-by: Jason Wang <jasowang@redhat.com>
>> >>>>       Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
>> >>>>
>> >>>> After this commit, win guest (winXP and win7) shows yellow
>> >>>> exclamation sign and is unable to start the device with
>> >>>> code 10.
>> >>>>
>> >>>> FWIW.  I'm not sure it is a good idea to make a release with
>> >>>> such a breakage, even rc0.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> /mjt
>> >
>> 

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Yan Vugenfirer]

On Feb 3, 2013, at 1:07 AM, Anthony Liguori wrote:

> Vadim Rozenfeld <vrozenfe@redhat.com> writes:
> 
>> On Sat, 2013-02-02 at 20:42 +0800, Jason Wang wrote:
>>> 
>>> Have a look at this issue. It was caused by multiqueue patch who adds a
>>> new field to virtio_net_cfg. Not sure multiqueue is the root cause since
>>> I also find even w/o multiqueue, adding any new field to virtio_net_cfg
>>> will break windows guest. Haven't had a clue on this, will continue
>>> investigate.
>> 
>> cc'ing Yan, our NDIS guy.
> 
> If it helps, mq changes the config size from 8 to 16 bytes.  If the
> driver was making an assumption about an 8-byte config size, that's
> likely what the problem is.
> 
> Regards,
> 
> Anthony Liguori

That's exactly the problem.

Best regards,
Yan.


> 
>> Thank you,
>> Vadim.
>>>> 
>>>>>> Regards,
>>>>>> 
>>>>>> Anthony Liguori
>>>>>> 
>>>>>>> commit fed699f9ca6ae8a0fb62803334cf46fa64d1eb91
>>>>>>> Author: Jason Wang <jasowang@redhat.com>
>>>>>>> Date:   Wed Jan 30 19:12:39 2013 +0800
>>>>>>> 
>>>>>>>      virtio-net: multiqueue support
>>>>>>> 
>>>>>>>      This patch implements both userspace and vhost support for multiple queue
>>>>>>>      virtio-net (VIRTIO_NET_F_MQ). This is done by introducing an array of
>>>>>>>      VirtIONetQueue to VirtIONet.
>>>>>>> 
>>>>>>>      Signed-off-by: Jason Wang <jasowang@redhat.com>
>>>>>>>      Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
>>>>>>> 
>>>>>>> After this commit, win guest (winXP and win7) shows yellow
>>>>>>> exclamation sign and is unable to start the device with
>>>>>>> code 10.
>>>>>>> 
>>>>>>> FWIW.  I'm not sure it is a good idea to make a release with
>>>>>>> such a breakage, even rc0.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> /mjt
>>>> 
>>> 
> 

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Michael Tokarev]

03.02.2013 17:23, Yan Vugenfirer wrote:

>> If it helps, mq changes the config size from 8 to 16 bytes.  If the
>> driver was making an assumption about an 8-byte config size, that's
>> likely what the problem is.
>>
> That's exactly the problem.

So what do we do?  It isn't nice to break existing setups.
Maybe mq can be disabled by default (together with Antony's
patch) until fixed drivers will be more widely available?
It's easy to turn off mq by default and turn it on when
needed...

The problem now is that it isn't obvious why your existing
setup breaks when you upgrade.  While when you especially
play with mq, you may look at options available around it...

How difficult it is to fix it in win driver?

Thanks,

/mjt

Re: [Qemu-devel] Headsup: windows virtio networking does not work on current git

[Anthony Liguori]

Michael Tokarev <mjt@tls.msk.ru> writes:

> 03.02.2013 17:23, Yan Vugenfirer wrote:
>
>>> If it helps, mq changes the config size from 8 to 16 bytes.  If the
>>> driver was making an assumption about an 8-byte config size, that's
>>> likely what the problem is.
>>>
>> That's exactly the problem.
>
> So what do we do?  It isn't nice to break existing setups.
> Maybe mq can be disabled by default (together with Antony's
> patch) until fixed drivers will be more widely available?

I've got a patch that does exactly like this.  It's pretty ugly though
because of the relationship between virtio-pci and virtio-net.  I'm
going to try to see if I can find a cleaner way to do it on Monday.

I'm also contemplating just disabling mq by default.  Since a special
command line is needed to enable it anyway (on the backend), having to
specify mq=on for the device doesn't seem so bad.

But yeah, we don't want Windows guests to break with -M pc by default so
we should do something to work around it.

N.B. this is a pretty nasty bug in the guest driver.  Any new virtio-net
feature is going to trigger it (not just multiqueue).  So while pc-1.3
will work forever with this guest image, it's pretty much guaranteed to
break at some point in the future.

> It's easy to turn off mq by default and turn it on when
> needed...
>
> The problem now is that it isn't obvious why your existing
> setup breaks when you upgrade.  While when you especially
> play with mq, you may look at options available around it...

If we ever to get virtio2, a really handy feature to have would be a
driver identification string.  Even if was only informative (and not
authoritative, we've had a lot of issues like this and being able to
make work arounds conditional on the driver identification string would
be nice.

Regards,

Anthony Liguori

>
> How difficult it is to fix it in win driver?
>
> Thanks,
>
> /mjt