[Bug 55421] New: igb VF can't work in KVM guest

[Bug 55421] New: igb VF can't work in KVM guest

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55421

           Summary: igb VF can't work in KVM guest
           Product: Virtualization
           Version: unspecified
    Kernel Version: 3.9.0-rc1
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: kvm
        AssignedTo: virtualization_kvm@kernel-bugs.osdl.org
        ReportedBy: yongjie.ren@intel.com
        Regression: No


Environment:
------------
Host OS (ia32/ia32e/IA64):ia32e
Guest OS (ia32/ia32e/IA64):ia32e
Guest OS Type (Linux/Windows):Linux
kvm.git next branch Commit:5da596078f915a62e39a20e582308eab91b88c9a
qemu-kvm uq/master branch Commit:3e41a753551a906dd9ed66fb0fc34167a6af3ba0
Host Kernel Version:3.9.0-rc1
Hardware:SNB-EP, WSM-EP


Bug detailed description:
--------------------------
After assigning a igb VF to a guest or hot-plugging a VF to a guest, the VF
cannot work in the guest.
Note:
1. The VF of Intel 82599 NIC (ixgbe) can work in guest
2. The PF of Intel 82576 NIC or Intel I350 NIC (using igb driver) can work in
guest

This should be a kvm bug.
kvm      + qemu-kvm   =  result
5da59607 + 3e41a753   = bad
3ab66e8a + 3e41a753   = good


Reproduce steps:
----------------
1.start up a KVM host
2.assign a VF to a guest in qemu commandline: 
qemu-system-x86_64 --enable-kvm -m 1024 -smp 2 -device pci-assign,host=05:10.0
-net none -hda rhel6u3.img

Current result:
----------------
the VF of cannot work in guest 

Expected result:
----------------
the VF can work in the guest

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55421] igb VF can't work in KVM guest

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55421





--- Comment #1 from Jay Ren <yongjie.ren@intel.com>  2013-03-18 15:11:16 ---
Created an attachment (id=95741)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=95741)
host dmesg

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55421] igb VF can't work in KVM guest

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55421





--- Comment #2 from Jay Ren <yongjie.ren@intel.com>  2013-03-18 15:12:02 ---
Created an attachment (id=95751)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=95751)
guest dmesg

in the dmesg in guest:
igbvf 0000:00:03.0: irq 26 for MSI/MSI-X
igbvf 0000:00:03.0: Invalid MAC Address: 00:00:00:00:00:00
igbvf: probe of 0000:00:03.0 failed with error -5

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55421] igb VF can't work in KVM guest

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55421





--- Comment #3 from Jay Ren <yongjie.ren@intel.com>  2013-03-18 15:12:33 ---
Created an attachment (id=95761)
 --> (https://bugzilla.kernel.org/attachment.cgi?id=95761)
lspci info of the igbvf in guest

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55421] igb VF can't work in KVM guest

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55421


Alex Williamson <alex.williamson@redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alex.williamson@redhat.com,
                   |                            |jeffrey.t.kirsher@intel.com




--- Comment #4 from Alex Williamson <alex.williamson@redhat.com>  2013-03-21 21:29:09 ---
Further bisected to:

commit 5ac6f91d39e0884813dc010e14552143cd1d0d8b
Author: Mitch A Williams <mitch.a.williams@intel.com>
Date:   Fri Jan 18 08:57:20 2013 +0000

    igb: Don't give VFs random MAC addresses

    If the user has not assigned a MAC address to a VM, then don't give it a
    random one. Instead, just give it zeros and let it figure out what to do
    with them.

    Signed-off-by: Mitch Williams <mitch.a.williams@intel.com>
    CC: Andy Gospodarek <andy@greyhouse.net>
    CC: Stefan Assmann <sassmann@kpanic.de>
    Tested-by: Aaron Brown <aaron.f.brown@intel.com>
    Tested-by: Stefan Assmann <sassmann@redhat.com>
    Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>

So, for whatever reason we no longer assign a random MAC address when using the
device in a VM (but we do still use one if attached to igbvf in the host).  I
expect we'll eventually see this on all the Intel SR-IOV NICs.  The solution is
to use the ip command to assign the VF a valid MAC address prior to using it
with KVM.  I'll let those who made the change defend it further if they wish.

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

Re: [Bug 55421] igb VF can't work in KVM guest

[Alex Williamson]

Mitch,

Bugzilla wouldn't let me add you to the CC, so FYI...
Thanks,

Alex

On Thu, 2013-03-21 at 21:29 +0000, bugzilla-daemon@bugzilla.kernel.org
wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=55421
> 
> 
> Alex Williamson <alex.williamson@redhat.com> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |alex.williamson@redhat.com,
>                    |                            |jeffrey.t.kirsher@intel.com
> 
> 
> 
> 
> --- Comment #4 from Alex Williamson <alex.williamson@redhat.com>  2013-03-21 21:29:09 ---
> Further bisected to:
> 
> commit 5ac6f91d39e0884813dc010e14552143cd1d0d8b
> Author: Mitch A Williams <mitch.a.williams@intel.com>
> Date:   Fri Jan 18 08:57:20 2013 +0000
> 
>     igb: Don't give VFs random MAC addresses
> 
>     If the user has not assigned a MAC address to a VM, then don't give it a
>     random one. Instead, just give it zeros and let it figure out what to do
>     with them.
> 
>     Signed-off-by: Mitch Williams <mitch.a.williams@intel.com>
>     CC: Andy Gospodarek <andy@greyhouse.net>
>     CC: Stefan Assmann <sassmann@kpanic.de>
>     Tested-by: Aaron Brown <aaron.f.brown@intel.com>
>     Tested-by: Stefan Assmann <sassmann@redhat.com>
>     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> 
> So, for whatever reason we no longer assign a random MAC address when using the
> device in a VM (but we do still use one if attached to igbvf in the host).  I
> expect we'll eventually see this on all the Intel SR-IOV NICs.  The solution is
> to use the ip command to assign the VF a valid MAC address prior to using it
> with KVM.  I'll let those who made the change defend it further if they wish.
> 

RE: [Bug 55421] igb VF can't work in KVM guest

[Williams, Mitch A]

Thanks for the FYI, Alex. When we pushed this patch upstream, there was a corresponding patch to igbvf that was pushed at the same time. With this patch, the VFs detect the zero address and generate a random MAC address themselves. The reason for this change is to make the VFs play nicely with udev. The issue was originally raised by Stefan Assmann.

With older drivers in the guest, this will cause the issue you noted. Distros should cherry-pick the corresponding VF patch.

-Mitch

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@redhat.com]
> Sent: Thursday, March 21, 2013 2:32 PM
> To: Williams, Mitch A
> Cc: kvm@vger.kernel.org
> Subject: Re: [Bug 55421] igb VF can't work in KVM guest
> 
> Mitch,
> 
> Bugzilla wouldn't let me add you to the CC, so FYI...
> Thanks,
> 
> Alex
> 
> On Thu, 2013-03-21 at 21:29 +0000, bugzilla-daemon@bugzilla.kernel.org
> wrote:
> > https://bugzilla.kernel.org/show_bug.cgi?id=55421
> >
> >
> > Alex Williamson <alex.williamson@redhat.com> changed:
> >
> >            What    |Removed                     |Added
> > --------------------------------------------------------------------------
> --
> >                  CC|
> |alex.williamson@redhat.com,
> >                    |
> |jeffrey.t.kirsher@intel.com
> >
> >
> >
> >
> > --- Comment #4 from Alex Williamson <alex.williamson@redhat.com>  2013-03-
> 21 21:29:09 ---
> > Further bisected to:
> >
> > commit 5ac6f91d39e0884813dc010e14552143cd1d0d8b
> > Author: Mitch A Williams <mitch.a.williams@intel.com>
> > Date:   Fri Jan 18 08:57:20 2013 +0000
> >
> >     igb: Don't give VFs random MAC addresses
> >
> >     If the user has not assigned a MAC address to a VM, then don't give it
> a
> >     random one. Instead, just give it zeros and let it figure out what to
> do
> >     with them.
> >
> >     Signed-off-by: Mitch Williams <mitch.a.williams@intel.com>
> >     CC: Andy Gospodarek <andy@greyhouse.net>
> >     CC: Stefan Assmann <sassmann@kpanic.de>
> >     Tested-by: Aaron Brown <aaron.f.brown@intel.com>
> >     Tested-by: Stefan Assmann <sassmann@redhat.com>
> >     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> >
> > So, for whatever reason we no longer assign a random MAC address when
> using the
> > device in a VM (but we do still use one if attached to igbvf in the host).
> I
> > expect we'll eventually see this on all the Intel SR-IOV NICs.  The
> solution is
> > to use the ip command to assign the VF a valid MAC address prior to using
> it
> > with KVM.  I'll let those who made the change defend it further if they
> wish.
> >
> 
> 

Re: [Bug 55421] igb VF can't work in KVM guest

[Alex Williamson]

On Thu, 2013-03-21 at 22:03 +0000, Williams, Mitch A wrote:
> Thanks for the FYI, Alex. When we pushed this patch upstream, there
> was a corresponding patch to igbvf that was pushed at the same time.
> With this patch, the VFs detect the zero address and generate a random
> MAC address themselves. The reason for this change is to make the VFs
> play nicely with udev. The issue was originally raised by Stefan
> Assmann.
> 
> With older drivers in the guest, this will cause the issue you noted.
> Distros should cherry-pick the corresponding VF patch.

I agree the random MAC doesn't play nicely with udev, although I have
found that my guest stops adding new persistent net rules after
eth680 ;)  Arguably anyone using VFs for more than testing should
already be assigning a stable MAC address, but I don't really think a
guest driver update across every possible guest OS is all that
practical.  At least that explains why igbvf in the host is unaffected
though.  Should we expect a Code 10 (device cannot start) from the
Windows igbvf driver when it finds a zero'd MAC?  Thanks,

Alex

> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@redhat.com]
> > Sent: Thursday, March 21, 2013 2:32 PM
> > To: Williams, Mitch A
> > Cc: kvm@vger.kernel.org
> > Subject: Re: [Bug 55421] igb VF can't work in KVM guest
> > 
> > Mitch,
> > 
> > Bugzilla wouldn't let me add you to the CC, so FYI...
> > Thanks,
> > 
> > Alex
> > 
> > On Thu, 2013-03-21 at 21:29 +0000, bugzilla-daemon@bugzilla.kernel.org
> > wrote:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=55421
> > >
> > >
> > > Alex Williamson <alex.williamson@redhat.com> changed:
> > >
> > >            What    |Removed                     |Added
> > > --------------------------------------------------------------------------
> > --
> > >                  CC|
> > |alex.williamson@redhat.com,
> > >                    |
> > |jeffrey.t.kirsher@intel.com
> > >
> > >
> > >
> > >
> > > --- Comment #4 from Alex Williamson <alex.williamson@redhat.com>  2013-03-
> > 21 21:29:09 ---
> > > Further bisected to:
> > >
> > > commit 5ac6f91d39e0884813dc010e14552143cd1d0d8b
> > > Author: Mitch A Williams <mitch.a.williams@intel.com>
> > > Date:   Fri Jan 18 08:57:20 2013 +0000
> > >
> > >     igb: Don't give VFs random MAC addresses
> > >
> > >     If the user has not assigned a MAC address to a VM, then don't give it
> > a
> > >     random one. Instead, just give it zeros and let it figure out what to
> > do
> > >     with them.
> > >
> > >     Signed-off-by: Mitch Williams <mitch.a.williams@intel.com>
> > >     CC: Andy Gospodarek <andy@greyhouse.net>
> > >     CC: Stefan Assmann <sassmann@kpanic.de>
> > >     Tested-by: Aaron Brown <aaron.f.brown@intel.com>
> > >     Tested-by: Stefan Assmann <sassmann@redhat.com>
> > >     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> > >
> > > So, for whatever reason we no longer assign a random MAC address when
> > using the
> > > device in a VM (but we do still use one if attached to igbvf in the host).
> > I
> > > expect we'll eventually see this on all the Intel SR-IOV NICs.  The
> > solution is
> > > to use the ip command to assign the VF a valid MAC address prior to using
> > it
> > > with KVM.  I'll let those who made the change defend it further if they
> > wish.
> > >
> > 

[Bug 55421] igb VF can't work in KVM guest

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55421


Jay Ren <yongjie.ren@intel.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID




--- Comment #5 from Jay Ren <yongjie.ren@intel.com>  2013-04-22 14:14:26 ---
As this is the latest igb drive change, I'll close this bug.

There are two methods to work around or fix this issue.
1. update igbvf driver or kernel of the guest
2. use ip command to set VF's MAC before assigning to a guest
(e.g. ip link set eth0 vf 0 mac 00:1E:67:65:93:01 )

-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.

[Bug 55421] igb VF can't work in KVM guest

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=55421


Jay Ren <yongjie.ren@intel.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |CLOSED




-- 
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are watching the assignee of the bug.