Re: KVM x86_64 with SR-IOV..?

[Sheng Yang <sheng@linux.intel.com>]

On Monday 04 May 2009 08:53:07 Nicholas A. Bellinger wrote:
> On Sat, 2009-05-02 at 18:22 +0800, Sheng Yang wrote:
> > On Thu, Apr 30, 2009 at 01:22:54PM -0700, Nicholas A. Bellinger wrote:
> > > Greetings KVM folks,
> > >
> > > I wondering if any information exists for doing SR-IOV on the new VT-d
> > > capable chipsets with KVM..?  From what I understand the patches for
> > > doing this with KVM are floating around, but I have been unable to find
> > > any user-level docs for actually making it all go against a upstream
> > > v2.6.30-rc3 code..
> > >
> > > So far I have been doing IOV testing with Xen 3.3 and 3.4.0-pre, and I
> > > am really hoping to be able to jump to KVM for single-function and and
> > > then multi-function SR-IOV.  I know that the VM migration stuff for IOV
> > > in Xen is up and running,  and I assume it is being worked in for KVM
> > > instance migration as well..?  This part is less important (at least
> > > for me :-) than getting a stable SR-IOV setup running under the KVM
> > > hypervisor..  Does anyone have any pointers for this..?
> > >
> > > Any comments or suggestions are appreciated!
> >
> > Hi Nicholas
> >
> > The patches are not floating around now. As you know, SR-IOV for Linux
> > have been in 2.6.30, so then you can use upstream KVM and qemu-kvm(or
> > recent released kvm-85) with 2.6.30-rc3 as host kernel. And some time
> > ago, there are several SRIOV related patches for qemu-kvm, and now they
> > all have been checked in.
> >
> > And for KVM, the extra document is not necessary, for you can simple
> > assign a VF to guest like any other devices. And how to create VF is
> > specific for each device driver. So just create a VF then assign it to
> > KVM guest is fine.
>
> Greetings Sheng,
>
> So, I have been trying the latest kvm-85 release on a v2.6.30-rc3
> checkout from linux-2.6.git on a CentOS 5u3 x86_64 install on Intel
> IOH-5520 based dual socket Nehalem board.  I have enabled DMAR and
> Interrupt Remapping my KVM host using v2.6.30-rc3 and from what I can
> tell, the KVM_CAP_* defines from libkvm are enabled with building kvm-85
> after './configure --kerneldir=/usr/src/linux-2.6.git' and the PCI
> passthrough code is being enabled in kvm-85/qemu/hw/device-assignment.c
> AFAICT..
>
> >From there, I use the freshly installed qemu-x86_64-system binary to
>
> start a Debian 5 x86_64 HVM (that previously had been moving network
> packets under Xen for PCIe passthrough). I see the MSI-X interrupt
> remapping working on the KVM host for the passed -pcidevice, and the
> MMIO mappings from the qemu build that I also saw while using
> Xen/qemu-dm built with PCI passthrough are there as well..
>

Hi Nicholas

> But while the KVM guest is booting, I see the following exception(s)
> from qemu-x86_64-system for one of the VFs for a multi-function PCIe
> device:
>
> BUG: kvm_destroy_phys_mem: invalid parameters (slot=-1)

This one is mostly harmless.
>
> I try with one of the on-board e1000e ports (02:00.0) and I see the same
> exception along with some MSI-X exceptions from qemu-x86_64-system in
> KVM guest.. However, I am still able to see the e1000e and the other
> vxge multi-function device with lspci, but I am unable to dhcp or ping
> with the e1000e and VF from multi-function device fails to register the
> MSI-X interrupt in the guest..

Did you see the interrupt in the guest and host side? I think you can try on-
board e1000e for MSI-X first. And please ensure correlated driver have been 
loaded correctly. And what do you mean by "some MSI-X exceptions"? Better with 
the log.
>
> Soooo, I enabled the debugging code in kvm-85/qemu/hw/device-assignment.c
> and see the PAGE aligned MMIO memory for the passed PCIe device is being
> released during the BUG exceptions above..  Is there something else I
> should be looking at..?  

That part of memory should be released for trap MMIO for MSI-X table.

> I have pci-stub enabled, and I unbind 02:00.0
> from /sys/bus/pci/drivers/e1000e/unbind successfully (just like with Xen
> and pciback), but I am unable to do the 'echo -n 02:00.0
>
> > /sys/bus/pci/drivers/pci-stub/bind' (it returns write error, no such
>
> device, with no dmesg output) on the KVM host running v2.6.30-rc3.  Is
> this supposed to happen on v2.6.30-rc3 with pci-stub..?  

Maybe you need "echo 0000:02:00.0 > /sys/bus/pci/drivers/pci-stub/bind"? 

> I am also using
> the the kvm-85 source dist kvm_intel.ko and kvm.ko kernel modules.  Is
> there something I am missing when building kvm-85 for SR-IOV passthrough..?

I think the first thing is to confirm that device assignment work in your 
environment, using on-board card. You can also refer to 
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

And you can post debug_device_assignment=1 log and qemu log and the tail of 
dmesg as well.

Thanks!

-- 
regards
Yang, Sheng

>
> Also FYI, I am having to use pci=resource_alignment=
> because the BIOS does not PAGE_SIZE align the MMIO BARs for my
> multi-function devices..
>
> Also, I tried with disabling the DMAR with the Intel IOMMU passthrough
> from this patch:
>
> https://lists.linux-foundation.org/pipermail/iommu/2009-April/001339.html
>
> that did not make it into v2.6.30-rc3.  The patch logic was enabled but
> still I saw the same kvm exceptions from qemu-system-x86_64.
>
> Anyways, I am going to give it a shot with the Fedora 11 x86_64 Preview
> and see if it works as expected with a IOH-5520 chipset with the AMI
> BIOS on a Tyan S7010 with Xeon 5520s.  Hopefully this is just a kvm-85
> build and/or install issue I am seeing on my CentOS 5u3 install (that has a
> Xen PCIe passthrough setup on it as well) with v2.6.30-rc3.  I will try on
> a fresh install on a distro with the new KVM logic and see what happens.
>
> :-)
>
> Thanks for your comments!
>
> --nab