Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

James Neave
2011-02-21T23:06:13Z

On Mon, Feb 21, 2011 at 10:55 PM, James Neave <roboj1m@gmail.com> wrote:
> On Mon, Feb 21, 2011 at 10:49 PM, James Neave <roboj1m@gmail.com> wrote:
>> On Mon, Feb 21, 2011 at 10:25 PM, James Neave <roboj1m@gmail.com> wrote:
>>> On Mon, Feb 21, 2011 at 9:01 PM, Alex Williamson
>>> <alex.williamson@redhat.com> wrote:
>>>> On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
>>>>> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
>>>>> <alex.williamson@redhat.com> wrote:
>>>>> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>>>>> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>>>>> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>>>>> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>>>>> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>>>>> >> >>>> Hi,
>>>>> >> >>>>
>>>>> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>>>>> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>>>>> >> >>>> 0000:08:06.2"
>>>>> >> >>>
>>>>> >> >>> This is a rather misleading error message. It is *expected* that
>>>>> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
>>>>> >> >>> error messages QEMU is printing aren't much help either, but
>>>>> >> >>> ultimately something is returning -EBUSY in the PCI device assign
>>>>> >> >>> step
>>>>> >> >>
>>>>> >> >> James, as far as I remember, I had the same issue when I set up my
>>>>> >> >> system. Looking at my current (working) boot-script, apparently I've
>>>>> >> >> added a 4th line which removes the pci-stub again as a
>>>>> >> >> workaround....and it works:
>>>>> >> >>
>>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>>>>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>>>>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>>>>> >> >>
>>>>> >> >> Best regards
>>>>> >> >> Kenni
>>>>> >> >>
>>>>> >> >
>>>>> >> > Hi Kenni,
>>>>> >> >
>>>>> >> > Can I get a bit more information on "boot-script" please? Which file
>>>>> >> > exaclty have you put this in? Did you write your own service script
>>>>> >> > and put it in init.d?
>>>>> >> >
>>>>> >> > I've tried this:
>>>>> >> >
>>>>> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>>>>> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>>>>> >> >
>>>>> >> > I'll try it again with the fourth line added, manually before I start the VM.
>>>>> >> >
>>>>> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>>>>> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>>>>> >> >
>>>>> >> > Obviously one looks up which driver is being used by the PCI id, but
>>>>> >> > how do I look up which driver my PCI card is using?
>>>>> >> >
>>>>> >> > Thanks,
>>>>> >> >
>>>>> >> > James.
>>>>> >> >
>>>>> >>
>>>>> >> Hi,
>>>>> >>
>>>>> >> OK, adding the fourth line does nothing, changing the second line to
>>>>> >> "driver" rather than "device" does nothing, including in combination
>>>>> >> with fourth line there/not there.
>>>>> >
>>>>> > Yep, the last line is just removing the id from pci-stub, it doesn't
>>>>> > change anything about devices that are already bound to it.  driver vs
>>>>> > device are just different ways to get to the same thing.
>>>>> >
>>>>> >> So I'm still stuck, can anybody else help?
>>>>> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
>>>>> >> against my kernel?
>>>>> >
>>>>> > I don't know why you're getting -EBUSY for this device, but maybe we can
>>>>> > start from a clean slate and see if it helps.  Here's what I would
>>>>> > suggest:
>>>>> >
>>>>> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
>>>>> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
>>>>> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
>>>>> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>>>>> >
>>>>> > Note we have to knock out the firewire because it shares an interrupt
>>>>> > with the ehci device you're trying to assign.  We want to remove the USB
>>>>> > controller entirely from the host.  Your dmesg indicates the host is
>>>>> > still seeing the device via the uhci ports, and isn't happy about it.
>>>>> > You can ignore pci-stub for the moment, it's just a way to keep drivers
>>>>> > from claiming the device, it's not required for device assignment.  Now,
>>>>> > instead of only trying to assign the ehci, let's move the whole usb
>>>>> > controller to the guest:
>>>>> >
>>>>> > -device pci-assign,host=08:06.0,addr=5.0 \
>>>>> > -device pci-assign,host=08:06.1,addr=5.1 \
>>>>> > -device pci-assign,host=08:06.2,addr=5.2
>>>>> >
>>>>> > (slot 5 on the guest is arbitrary, pick something else if you need to)
>>>>> > If that works, then you can bind all those devices to pci-stub and it
>>>>> > should still work.
>>>>> >
>>>>> > Alex
>>>>>
>>>>> Hi,
>>>>>
>>>>> Sorry about the slow reply, I hosed all of my PCs fiddling with
>>>>> compiling the latest qemu, took me a while to put it all back together
>>>>> again in between work!
>>>>>
>>>>> I'm afraid I use virtmanager, although I guess using the "add pci
>>>>> device" function is the same as -device pci-assign? It does seem to
>>>>> add it to the command line that gets written out to the log files in
>>>>> /var/log/libvirt/qemu.
>>>>>
>>>>> Nevertheless, I've tried that and still not luck, the log output is
>>>>> similar, with one extra line:
>>>>>
>>>>> http://pastebin.com/MJ6aqjNq
>>>>
>>>> You actually ended up with:
>>>>
>>>> -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
>>>> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
>>>> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>>>>
>>>> Which isn't quite what I was suggesting.  You probably have xml that
>>>> looks like this:
>>>>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>
>>>>
>>>>
>>>>    </hostdev>
>>>>    ...
>>>>
>>>> Try adding an address line, so you get something more like this:
>>>>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>
>>>>
>>>>
>>>>
>>>>    </hostdev>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>
>>>>
>>>>
>>>>
>>>>    </hostdev>
>>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>>
>>>>
>>>>
>>>>
>>>>    </hostdev>
>>>>
>>>>
>>>>> Using raw in/out ioport access (sysfs - Input/output error)
>>>>
>>>> This is just an informational line to let us know whether pci-sysfs
>>>> supports read/write on the ioport resource files.  If it does, we use
>>>> those rather than doing raw in/out for access to the device.  This does
>>>> highlight another potential problem.  Your distro probably doesn't have
>>>> all the patches in place for non-privileged device assignment, which
>>>> could be why you're having strange issues.  Check
>>>> your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
>>>> they're not already, try setting them to 'root', restart libvirtd and
>>>> see if anything improves.
>>>>
>>>>> Here's the dmesg output:
>>>>> http://pastebin.com/AE1euUN1
>>>>
>>>> If still issues after the above, it might be useful to pastbin the
>>>> entire dmesg so we can make sure the iommu is really on.  Thanks,
>>>>
>>>> Alex
>>>
>>> Hi,
>>>
>>> No such luck I'm afraid.
>>>
>>> Here is the original XML:
>>>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>
>>>
>>>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>
>>>
>>>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
>>> function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>
>>>
>>>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
>>> function='0x0'/>
>>>    </hostdev>
>>>
>>> The only difference from your recommended change is the target (I take
>>> it they are target addresses for the VM?) addresses run:
>>> 0000:00:06.0
>>> 0000:00:07.0
>>> 0000:00:08.0
>>>
>>> Instead of:
>>> 0000:00:06.0
>>> 0000:00:06.1
>>> 0000:00:06.2
>>>
>>> Regardless, I still changed test.xml to:
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>>
>>>
>>>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x0'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>
>>>
>>>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x1'/>
>>>    </hostdev>
>>>    <hostdev mode='subsystem' type='pci' managed='yes'>
>>>
>>>
>>>
>>>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
>>> function='0x2'/>
>>>    </hostdev>
>>>
>>> To no effect.
>>>
>>> In qemu.conf, user and group were commented out, I uncommented both
>>> and they were both already set to root.
>>>
>>> After both a restart of libvirt-bin and the pc itself, no change.
>>>
>>> Finally, here is the very latest dmesg:
>>> http://pastebin.com/9HE61K62
>>>
>>> Does anybody know the debug kernel switches for iommu?
>>>
>>> Many Thanks,
>>>
>>> James.
>>>
>>
>> Hi,
>>
>> A ray of sunshine (ish)
>>
>> I can apparently pass through this device:
>>
>> 00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA) (rev 40)
>>        Subsystem: Giga-byte Technology Device a102
>>        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
>> ParErr- Stepping- SERR- FastB2B- DisINTx-
>>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort-
>> <TAbort- <MAbort- >SERR- <PERR- INTx-
>>        Latency: 32, Cache Line Size: 64 bytes
>>        Interrupt: pin ? routed to IRQ 16
>>        Region 0: Memory at fdff4000 (64-bit, non-prefetchable) [size=16K]
>>        Capabilities: <access denied>
>>        Kernel driver in use: HDA Intel
>>        Kernel modules: snd-hda-intel
>>
>> From test.log:
>>
>> 2011-02-21 22:40:58.888: starting up
>> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
>> QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp
>> 3,sockets=3,cores=1,threads=1 -name test -uuid
>> 307bfcd2-9dec-29b7-1b4d-c46cd9d7cdbc -nodefconfig -nodefaults -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=readline -rtc base=utc -boot
>> c -drive file=/var/lib/libvirt/images/test.img,if=none,id=drive-virtio-disk0,boot=on,format=raw
>> -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
>> -drive file=/home/james/ubuntu_10.10_x86.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
>> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
>> -netdev tap,fd=59,id=hostnet0 -device
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -usb -vnc 127.0.0.1:1 -vga
>> cirrus -device pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
>> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
>> char device redirected to /dev/pts/1
>> kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7d:32:7c,bus=pci.0,addr=0x3:
>> pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
>> 2011-02-21 22:41:48.090: shutting down
>>
>> Specifically: -device
>> pci-assign,host=00:14.2,id=hostdev0,configfd=60,bus=pci.0,addr=0x6
>>
>> Unfortunately I can't get it to boot an ISO through virtmanager! ><
>> Well, not without deleting the VM and recreating it again.
>> I shall try getting the latest virt-manager.
>>
>> Regards,
>>
>> James.
>>
>
> Or I could just try pressing the Apply button... ¬.¬;;
>
> J.
>

OK, it doesn't work but here's the dmesg from the test vm with the
host's on board sound passed through:

http://pastebin.com/N2bdMQTa

Specifically the error is:

[ 13.795117] hda-intel: unable to grab IRQ 0, disabling device
[ 13.797976] HDA Intel: probe of 0000:00:06.0 failed with error -16

Regards,

James.