kvm pci passthrough

kvm pci passthrough

[David Brown]

I was wondering what the status of kvm pci passthrough is? Especially
the kind where I don't have vt-d? Where are the parts to play with? Is
there any documentation you can point me to?

Thanks,
- David Brown

Re: kvm pci passthrough

[Muli Ben-Yehuda]

On Wed, Oct 29, 2008 at 05:44:54PM -0700, David Brown wrote:

> I was wondering what the status of kvm pci passthrough is?
> Especially the kind where I don't have vt-d? Where are the parts to
> play with? Is there any documentation you can point me to?

Hi David,

The kind where you *do* have VT-d is now included in the kernel and
userspace trees, and should "just work" (tell us if it doesn't). You
should clone the kernel and userspace trees, verify that the device
assignment patches are included in the userspace tree (they were
applied just now and may take some time to sync) and then run the
guest with:

-pcidevice host=bus:dev.func

Where the bus, dev, and func are what lspci shows you for the device
(on the host). You should also make sure that the device isn't used by
any driver on the host first.

For the other kinds (pvdma-without-iommu and reserved mem) I don't
know of the latest status.

Cheers,
Muli
-- 
The First Workshop on I/O Virtualization (WIOV '08)
Dec 2008, San Diego, CA, http://www.usenix.org/wiov08/
                       <->
SYSTOR 2009---The Israeli Experimental Systems Conference
http://www.haifa.il.ibm.com/conferences/systor2009/

Re: kvm pci passthrough

[David Brown]

On Thu, Oct 30, 2008 at 5:26 AM, Muli Ben-Yehuda <muli@il.ibm.com> wrote:
> On Wed, Oct 29, 2008 at 05:44:54PM -0700, David Brown wrote:
>
>> I was wondering what the status of kvm pci passthrough is?
>> Especially the kind where I don't have vt-d? Where are the parts to
>> play with? Is there any documentation you can point me to?
>
> Hi David,
>
> The kind where you *do* have VT-d is now included in the kernel and
> userspace trees, and should "just work" (tell us if it doesn't). You
> should clone the kernel and userspace trees, verify that the device
> assignment patches are included in the userspace tree (they were
> applied just now and may take some time to sync) and then run the
> guest with:

Thanks, glad to know progress is being made. However, around here
we're mostly an AMD shop and as far as I know they don't have a
working IOMMU yet. Has there been any testing with High Performance
Networking with RDMA support like infiniband?

> -pcidevice host=bus:dev.func
>
> Where the bus, dev, and func are what lspci shows you for the device
> (on the host). You should also make sure that the device isn't used by
> any driver on the host first.

I assume the pci passthrough without IOMMU will also have the same
interface? Are there commands in the qemu monitor to pass these in on
a running guest?

> For the other kinds (pvdma-without-iommu and reserved mem) I don't
> know of the latest status.

Hmm, Okay thanks for the info.

- David Brown

Re: kvm pci passthrough

[Muli Ben-Yehuda]

On Thu, Oct 30, 2008 at 08:42:50AM -0700, David Brown wrote:

> Thanks, glad to know progress is being made. However, around here
> we're mostly an AMD shop and as far as I know they don't have a
> working IOMMU yet.

I believe it's working, just not shipping yet :-)

> Has there been any testing with High Performance Networking with
> RDMA support like infiniband?

No, as far as I know.

> > -pcidevice host=bus:dev.func
> >
> > Where the bus, dev, and func are what lspci shows you for the device
> > (on the host). You should also make sure that the device isn't used by
> > any driver on the host first.
> 
> I assume the pci passthrough without IOMMU will also have the same
> interface?

I would assume so too.

> Are there commands in the qemu monitor to pass these in on a running
> guest?

Yes, via an extension of the pci_add monitor command to take a host
BDF argument.

Cheers,
Muli

-- 
The First Workshop on I/O Virtualization (WIOV '08)
Dec 2008, San Diego, CA, http://www.usenix.org/wiov08/
                       <->
SYSTOR 2009---The Israeli Experimental Systems Conference
http://www.haifa.il.ibm.com/conferences/systor2009/

Re: kvm pci passthrough

[Chris Jones]

Muli,

> The kind where you *do* have VT-d is now included in the kernel and
> userspace trees, and should "just work" (tell us if it doesn't). You
> should clone the kernel and userspace trees, verify that the device
> assignment patches are included in the userspace tree (they were
> applied just now and may take some time to sync) and then run the
> guest with:
> 

I'm also interested in PCI passthrough support.  

Do you have any estimation when this support will be included in the kernel.org
and the kvm-nn releases on sourceforge?  Or will it be only available under git
trees for the foreseeable future?

Thanks,
Chris

Re: kvm pci passthrough

[Amit Shah]

* On Wednesday 05 Nov 2008 03:50:17 Chris Jones wrote:
> Muli,
>
> > The kind where you *do* have VT-d is now included in the kernel and
> > userspace trees, and should "just work" (tell us if it doesn't). You
> > should clone the kernel and userspace trees, verify that the device
> > assignment patches are included in the userspace tree (they were
> > applied just now and may take some time to sync) and then run the
> > guest with:
>
> I'm also interested in PCI passthrough support.
>
> Do you have any estimation when this support will be included in the
> kernel.org and the kvm-nn releases on sourceforge?  Or will it be only
> available under git trees for the foreseeable future?

The 2.6.28 kernel will have KVM device assignment support. The next kvm 
release (which is kvm-79) will have device assignment support both in the 
kernel and userspace components.

Amit.

KVM & PCI Passthrough

[Phil Daws]

Hello all, 

as performing some tests in my lab with PCI pass-through and not really understanding it.  Here is what I have done so far:

lspci -nn:

06:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] [1000:0079] (rev 05)

modprobe pci_stub
echo "1000 0079" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:06:00.0" > /sys/bus/pci/devices/0000:00:1b.0/driver/unbind
echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind

I then create the new guest through virt-manager, add the PCI device, but when it tries to start the install it fails straight away.  I have checked then IOMMU is enabled:

dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled

am running a Intel i7 4440k on a ASUS Z87-K Mobo with virtualization all enabled.

Any help would appreciated as wish to test http://www.osnexus.com/storage-appliance-os/ under KVM with a RAID card directly associated to the guest.

Thanks.

Re: KVM & PCI Passthrough

[Alex Williamson]

On Wed, 2013-12-04 at 18:08 +0000, Phil Daws wrote:
> Hello all, 
> 
> as performing some tests in my lab with PCI pass-through and not really understanding it.  Here is what I have done so far:
> 
> lspci -nn:
> 
> 06:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] [1000:0079] (rev 05)
> 
> modprobe pci_stub
> echo "1000 0079" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:06:00.0" > /sys/bus/pci/devices/0000:00:1b.0/driver/unbind
> echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> 
> I then create the new guest through virt-manager, add the PCI device, but when it tries to start the install it fails straight away.  I have checked then IOMMU is enabled:
> 
> dmesg | grep IOMMU
> [    0.000000] Intel-IOMMU: enabled

This is not nearly enough output, my properly working Haswell i5-4430
reports:

$ dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.022767] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.023012] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.023402] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.355789] IOMMU 0 0xfed90000: using Queued invalidation
[    0.355878] IOMMU 1 0xfed91000: using Queued invalidation
[    0.355968] IOMMU: Setting RMRR:
[    0.356063] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]
[    0.357644] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc979000 - 0xbc985fff]
[    0.357817] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc979000 - 0xbc985fff]
[    0.357981] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc979000 - 0xbc985fff]
[    0.358139] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.358234] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Make sure VT-d is enabled in the BIOS and boot with intel_iommu=on.

> am running a Intel i7 4440k on a ASUS Z87-K Mobo with virtualization all enabled.

Double check your CPU mode, 4440K doesn't exist according to
http://ark.intel.com/  The unlocked K models often (typically) do not
support VT-d, in which case you're out of luck with this CPU.  Thanks,

Alex

Re: KVM & PCI Passthrough

[Phil Daws]

----- Original Message -----
From: "Alex Williamson" <alex.williamson@redhat.com>
To: "Phil Daws" <uxbod@splatnix.net>
Cc: kvm@vger.kernel.org
Sent: Wednesday, 4 December, 2013 6:46:18 PM
Subject: Re: KVM & PCI Passthrough

On Wed, 2013-12-04 at 18:08 +0000, Phil Daws wrote:
> Hello all, 
> 
> as performing some tests in my lab with PCI pass-through and not really understanding it.  Here is what I have done so far:
> 
> lspci -nn:
> 
> 06:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] [1000:0079] (rev 05)
> 
> modprobe pci_stub
> echo "1000 0079" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:06:00.0" > /sys/bus/pci/devices/0000:00:1b.0/driver/unbind
> echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> 
> I then create the new guest through virt-manager, add the PCI device, but when it tries to start the install it fails straight away.  I have checked then IOMMU is enabled:
> 
> dmesg | grep IOMMU
> [    0.000000] Intel-IOMMU: enabled

This is not nearly enough output, my properly working Haswell i5-4430
reports:

$ dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.022767] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.023012] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.023402] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.355789] IOMMU 0 0xfed90000: using Queued invalidation
[    0.355878] IOMMU 1 0xfed91000: using Queued invalidation
[    0.355968] IOMMU: Setting RMRR:
[    0.356063] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]
[    0.357644] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc979000 - 0xbc985fff]
[    0.357817] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc979000 - 0xbc985fff]
[    0.357981] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc979000 - 0xbc985fff]
[    0.358139] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.358234] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Make sure VT-d is enabled in the BIOS and boot with intel_iommu=on.

> am running a Intel i7 4440k on a ASUS Z87-K Mobo with virtualization all enabled.

Double check your CPU mode, 4440K doesn't exist according to
http://ark.intel.com/  The unlocked K models often (typically) do not
support VT-d, in which case you're out of luck with this CPU.  Thanks,

Alex
---
Alex, thanks. Yeah, it is a i7-4770k, and I did set the grub line:

grep iommu /etc/default/grub
GRUB_CMDLINE_LINUX="intel_iommu=on"

Darn, http://ark.intel.com/products/75123/Intel-Core-i7-4770K-Processor-8M-Cache-up-to-3_90-GHz?q=4770k looks like am out of luck :(

Oh well, will just set up some storage from the host and pass that through to the guest.

Thank you, and one I shall watch for next time.