PCIe device pass-through - No IOMMU, Failed to deassign device error

PCIe device pass-through - No IOMMU, Failed to deassign device error

[Yigal Korman]

Hi,
I'm trying to pass a second video card to a Windows 7 virtual machine
with KVM, and I get the following error:
"
root@ubuntu-desktop:~# kvm -cpu qemu64 -hda /dev/sdb -cdrom /dev/cdrom
-boot order=dc -m 2000 -usb -name Win7x64 -enable-kvm -device
pci-assign,host=80:00.0
No IOMMU found.  Unable to assign device "(null)"
Failed to deassign device "(null)" : Invalid argument
Error initializing device pci-assign
"
Now it look like I don't have VT-d, but I do, here is my cpuinfo:
"
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU           E5440  @ 2.83GHz
stepping : 10
cpu MHz : 1998.000
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64
monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave
lahf_lm tpr_shadow vnmi flexpriority
bogomips : 5667.49
clflush size : 64
cache_alignment : 64
address sizes : 38 bits physical, 48 bits virtual
power management:
" ... this continues until processor reaches 7 (dual Xeon quad core)
I've enabled vt-d in the BIOS, and added this parameter to the kernel:
"intel_iommu=on"
I've ran these to unbind the card from the host OS:
"
modprobe pci_stub
echo "10de 040f" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:80:00.0 > /sys/bus/pci/devices/0000\:80\:00.0/driver/unbind
echo 0000:80:00.0 > /sys/bus/pci/drivers/pci-stub/bind
"
this is the relevant part from lspci -vv (I have two video cards, one
I'd like for the host and one for the guest):
"
60:00.0 VGA compatible controller: nVidia Corporation G84 [Quadro FX
1700] (rev a1)
Subsystem: nVidia Corporation Device 049a
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 28
Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at a0000000 (64-bit, prefetchable) [size=512M]
Region 3: Memory at f4000000 (64-bit, non-prefetchable) [size=32M]
Region 5: I/O ports at 2000 [size=128]
Capabilities: [60] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
Address: 0000000000000000  Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #8, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
Capabilities: [100] Virtual Channel <?>
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information <?>
Kernel driver in use: nvidia
Kernel modules: nvidia, nvidiafb
80:00.0 VGA compatible controller: nVidia Corporation G84 [Quadro FX
1700] (rev a1)
Subsystem: nVidia Corporation Device 049a
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 24
Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at c0000000 (64-bit, prefetchable) [size=512M]
Region 3: Memory at f0000000 (64-bit, non-prefetchable) [size=32M]
Region 5: I/O ports at 1000 [size=128]
Capabilities: [60] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
Address: 0000000000000000  Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #4, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive-
BWMgmt- ABWMgmt-
Capabilities: [100] Virtual Channel <?>
Capabilities: [128] Power Budgeting <?>
Capabilities: [600] Vendor Specific Information <?>
Kernel driver in use: pci-stub
Kernel modules: nvidia, nvidiafb
"
I'm using packages from Ubuntu Lucid (10.04 alpha) [kernel
2.6.32.11-11, qemu-kvm 0.12.2-ubuntu1]
I would really appreciate your help.

Thank you,
Yigal Korman.

PS my original mail was returned because it contained html, trying to
send it again with plain text.
--
Due to the recession, to save on energy costs, the light at the end of
the tunnel will be turned off.
—God

Re: PCIe device pass-through - No IOMMU, Failed to deassign device error

[Brian Jackson]

On Saturday 23 January 2010 05:20:49 Yigal Korman wrote:
> Hi,
> I'm trying to pass a second video card to a Windows 7 virtual machine
> with KVM, and I get the following error:


KVM doesn't support assigning graphics cards to VMs yet. There are people 
working on it afaik, but I don't know the progress.


> "
> root@ubuntu-desktop:~# kvm -cpu qemu64 -hda /dev/sdb -cdrom /dev/cdrom
> -boot order=dc -m 2000 -usb -name Win7x64 -enable-kvm -device
> pci-assign,host=80:00.0
> No IOMMU found.  Unable to assign device "(null)"
> Failed to deassign device "(null)" : Invalid argument
> Error initializing device pci-assign
> "
> Now it look like I don't have VT-d, but I do, here is my cpuinfo:
> "
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 23
> model name : Intel(R) Xeon(R) CPU           E5440  @ 2.83GHz
> stepping : 10
> cpu MHz : 1998.000
> cache size : 6144 KB
> physical id : 0
> siblings : 4
> core id : 0
> cpu cores : 4
> apicid : 0
> initial apicid : 0
> fpu : yes
> fpu_exception : yes
> cpuid level : 13
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
> pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
> lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64
> monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave
> lahf_lm tpr_shadow vnmi flexpriority
> bogomips : 5667.49
> clflush size : 64
> cache_alignment : 64
> address sizes : 38 bits physical, 48 bits virtual
> power management:
> " ... this continues until processor reaches 7 (dual Xeon quad core)
> I've enabled vt-d in the BIOS, and added this parameter to the kernel:
> "intel_iommu=on"
> I've ran these to unbind the card from the host OS:
> "
> modprobe pci_stub
> echo "10de 040f" > /sys/bus/pci/drivers/pci-stub/new_id
> echo 0000:80:00.0 > /sys/bus/pci/devices/0000\:80\:00.0/driver/unbind
> echo 0000:80:00.0 > /sys/bus/pci/drivers/pci-stub/bind
> "
> this is the relevant part from lspci -vv (I have two video cards, one
> I'd like for the host and one for the guest):
> "
> 60:00.0 VGA compatible controller: nVidia Corporation G84 [Quadro FX
> 1700] (rev a1)
> Subsystem: nVidia Corporation Device 049a
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Interrupt: pin A routed to IRQ 28
> Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
> Region 1: Memory at a0000000 (64-bit, prefetchable) [size=512M]
> Region 3: Memory at f4000000 (64-bit, non-prefetchable) [size=32M]
> Region 5: I/O ports at 2000 [size=128]
> Capabilities: [60] Power Management version 2
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
>  Enable- Address: 0000000000000000  Data: 0000
> Capabilities: [78] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
> LnkCap: Port #8, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1
>  <4us ClockPM- Suprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive-
> BWMgmt- ABWMgmt-
> Capabilities: [100] Virtual Channel <?>
> Capabilities: [128] Power Budgeting <?>
> Capabilities: [600] Vendor Specific Information <?>
> Kernel driver in use: nvidia
> Kernel modules: nvidia, nvidiafb
> 80:00.0 VGA compatible controller: nVidia Corporation G84 [Quadro FX
> 1700] (rev a1)
> Subsystem: nVidia Corporation Device 049a
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
> Stepping- SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Interrupt: pin A routed to IRQ 24
> Region 0: Memory at f2000000 (32-bit, non-prefetchable) [size=16M]
> Region 1: Memory at c0000000 (64-bit, prefetchable) [size=512M]
> Region 3: Memory at f0000000 (64-bit, non-prefetchable) [size=32M]
> Region 5: I/O ports at 1000 [size=128]
> Capabilities: [60] Power Management version 2
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [68] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
>  Enable- Address: 0000000000000000  Data: 0000
> Capabilities: [78] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <4us
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
> RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 128 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
> LnkCap: Port #4, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1
>  <4us ClockPM- Suprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive-
> BWMgmt- ABWMgmt-
> Capabilities: [100] Virtual Channel <?>
> Capabilities: [128] Power Budgeting <?>
> Capabilities: [600] Vendor Specific Information <?>
> Kernel driver in use: pci-stub
> Kernel modules: nvidia, nvidiafb
> "
> I'm using packages from Ubuntu Lucid (10.04 alpha) [kernel
> 2.6.32.11-11, qemu-kvm 0.12.2-ubuntu1]
> I would really appreciate your help.
> 
> Thank you,
> Yigal Korman.
> 
> PS my original mail was returned because it contained html, trying to
> send it again with plain text.
> --
> Due to the recession, to save on energy costs, the light at the end of
> the tunnel will be turned off.
> —God
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: PCIe device pass-through - No IOMMU, Failed to deassign device error

[Chris Wright]

* Brian Jackson (iggy@theiggy.com) wrote:
> On Saturday 23 January 2010 05:20:49 Yigal Korman wrote:
> > I'm trying to pass a second video card to a Windows 7 virtual machine
> > with KVM, and I get the following error:
> 
> KVM doesn't support assigning graphics cards to VMs yet. There are people 
> working on it afaik, but I don't know the progress.

Right, so even if you figure out the issue below, there's still issue w/
the actual graphcis device funtioning properly in the guest.

> > root@ubuntu-desktop:~# kvm -cpu qemu64 -hda /dev/sdb -cdrom /dev/cdrom
> > -boot order=dc -m 2000 -usb -name Win7x64 -enable-kvm -device
> > pci-assign,host=80:00.0
> > No IOMMU found.  Unable to assign device "(null)"
> > Failed to deassign device "(null)" : Invalid argument
> > Error initializing device pci-assign
> > "
> > Now it look like I don't have VT-d, but I do, here is my cpuinfo:
> > "

VT-d is a chipset feature, not a CPU feature.

<snip>
> > I've enabled vt-d in the BIOS, and added this parameter to the kernel:
> > "intel_iommu=on"

Again, VT (or VT-x) isn't the same as VT-d.  So to be sure, you can
grep dmesg for DMAR and IOMMU to verify that the chipset actually has
VT-d support, that it's enabled, and that it's not broken (there are
quite a few broken BIOS out there that case the IOMMU to be unusable).

thanks,
-chris

Re: PCIe device pass-through - No IOMMU, Failed to deassign device error

[Kenni Lund]

2010/1/26 Chris Wright <chrisw@sous-sol.org>:
>
> Again, VT (or VT-x) isn't the same as VT-d.  So to be sure, you can
> grep dmesg for DMAR and IOMMU to verify that the chipset actually has
> VT-d support, that it's enabled, and that it's not broken (there are
> quite a few broken BIOS out there that case the IOMMU to be unusable).

dmesg | egrep (DMAR|IOMMU)
This information should _really_ be added to the wiki at
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

Knowing this, it's quite easy for a user to determine if his system
has VT-d support, _before_ following the guide, compiling own kernel,
setting up qemu-kvm, unbinding and rebinding PCI devices, just to have
qemu-kvm 0.12.2 tell him that the system has no IOMMU (much better
than 0.12.1, agreed, but it's a bit late in the process to find out
:))

Can someone with write permissions to the wiki please add this?

Best Regards
Kenni Lund

Re: PCIe device pass-through - No IOMMU, Failed to deassign device error

[Kenni Lund]

2010/1/26 Kenni Lund <kenni@kelu.dk>:
> 2010/1/26 Chris Wright <chrisw@sous-sol.org>:
>>
>> Again, VT (or VT-x) isn't the same as VT-d.  So to be sure, you can
>> grep dmesg for DMAR and IOMMU to verify that the chipset actually has
>> VT-d support, that it's enabled, and that it's not broken (there are
>> quite a few broken BIOS out there that case the IOMMU to be unusable).
>
> dmesg | egrep (DMAR|IOMMU)
> This information should _really_ be added to the wiki at
> http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
>
> Knowing this, it's quite easy for a user to determine if his system
> has VT-d support, _before_ following the guide, compiling own kernel,
> setting up qemu-kvm, unbinding and rebinding PCI devices, just to have
> qemu-kvm 0.12.2 tell him that the system has no IOMMU (much better
> than 0.12.1, agreed, but it's a bit late in the process to find out
> :))

Doh, I didn't consider if the kernel compilation probably were needed
to give any output - nevertheless, I still think this should be added
to the wiki, even if it's the case. Perhaps a short text describing
what you should look for.

Best Regards
Kenni Lund

Re: PCIe device pass-through - No IOMMU, Failed to deassign device error

[Chris Wright]

* Kenni Lund (kenni@kelu.dk) wrote:
> 2010/1/26 Kenni Lund <kenni@kelu.dk>:
> > 2010/1/26 Chris Wright <chrisw@sous-sol.org>:
> >>
> >> Again, VT (or VT-x) isn't the same as VT-d.  So to be sure, you can
> >> grep dmesg for DMAR and IOMMU to verify that the chipset actually has
> >> VT-d support, that it's enabled, and that it's not broken (there are
> >> quite a few broken BIOS out there that case the IOMMU to be unusable).
> >
> > dmesg | egrep (DMAR|IOMMU)
> > This information should _really_ be added to the wiki at
> > http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
> >
> > Knowing this, it's quite easy for a user to determine if his system
> > has VT-d support, _before_ following the guide, compiling own kernel,
> > setting up qemu-kvm, unbinding and rebinding PCI devices, just to have
> > qemu-kvm 0.12.2 tell him that the system has no IOMMU (much better
> > than 0.12.1, agreed, but it's a bit late in the process to find out
> > :))
> 
> Doh, I didn't consider if the kernel compilation probably were needed
> to give any output - nevertheless, I still think this should be added
> to the wiki, even if it's the case. Perhaps a short text describing
> what you should look for.

Sure, I added a short snippet.

thanks,
-chris

Re: PCIe device pass-through - No IOMMU, Failed to deassign device error

[Avi Kivity]

On 01/26/2010 03:11 AM, Kenni Lund wrote:
>
> Can someone with write permissions to the wiki please add this?
>    

Everyone has write permissions, you just need an account.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: PCIe device pass-through - No IOMMU, Failed to deassign device error

[Yigal Korman]

Hi,
Thank you for the responses.
I've managed to get further in the process while running KVM with
Fedora rather than Ubuntu.
I've gotten Windows 7 to recognize the graphics card but mark it with
an error about hardware resources (I assume memory/interrupts).
I've found the same error was encountered while trying to achieve this
goal using Xen, here is a blog which claims to succeed overcoming this
issue and achieving GPU pass-through with an NVIDIA GForce card with
Xen.
Maybe it can help your progress on the matter.

Thanks again,
Yigal.

On Tue, Jan 26, 2010 at 08:23, Avi Kivity <avi@redhat.com> wrote:
> On 01/26/2010 03:11 AM, Kenni Lund wrote:
>>
>> Can someone with write permissions to the wiki please add this?
>>
>
> Everyone has write permissions, you just need an account.
>
> --
> Do not meddle in the internals of kernels, for they are subtle and quick to
> panic.
>
>



-- 
Due to the recession, to save on energy costs, the light at the end of
the tunnel will be turned off.
—God