Re: PCI Passthrough, error: The driver 'pci-stub' is occupying your device 0000:08:06.2

[James Neave <roboj1m@gmail.com>]

On Mon, Feb 21, 2011 at 9:01 PM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> On Mon, 2011-02-21 at 20:31 +0000, James Neave wrote:
>> On Mon, Feb 14, 2011 at 5:48 PM, Alex Williamson
>> <alex.williamson@redhat.com> wrote:
>> > On Sat, 2011-02-12 at 16:04 +0000, James Neave wrote:
>> >> On Tue, Feb 8, 2011 at 10:17 AM, James Neave <roboj1m@gmail.com> wrote:
>> >> > On Tue, Feb 8, 2011 at 9:59 AM, Kenni Lund <kenni@kelu.dk> wrote:
>> >> >> 2011/2/7 Daniel P. Berrange <berrange@redhat.com>:
>> >> >>> On Sat, Feb 05, 2011 at 04:34:01PM +0000, James Neave wrote:
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> I'm trying to pass a NOVA-T-500 TV Tuner card through to a gust VM.
>> >> >>>> I'm getting the error "The driver 'pci-stub' is occupying your device
>> >> >>>> 0000:08:06.2"
>> >> >>>
>> >> >>> This is a rather misleading error message. It is *expected* that
>> >> >>> pci-stub will occupy the device. Unfortunately the rest of the
>> >> >>> error messages QEMU is printing aren't much help either, but
>> >> >>> ultimately something is returning -EBUSY in the PCI device assign
>> >> >>> step
>> >> >>
>> >> >> James, as far as I remember, I had the same issue when I set up my
>> >> >> system. Looking at my current (working) boot-script, apparently I've
>> >> >> added a 4th line which removes the pci-stub again as a
>> >> >> workaround....and it works:
>> >> >>
>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/new_id
>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/ivtv/unbind
>> >> >> echo "0000:04:08.0" > /sys/bus/pci/drivers/pci-stub/bind
>> >> >> echo "4444 0016" > /sys/bus/pci/drivers/pci-stub/remove_id
>> >> >>
>> >> >> Best regards
>> >> >> Kenni
>> >> >>
>> >> >
>> >> > Hi Kenni,
>> >> >
>> >> > Can I get a bit more information on "boot-script" please? Which file
>> >> > exaclty have you put this in? Did you write your own service script
>> >> > and put it in init.d?
>> >> >
>> >> > I've tried this:
>> >> >
>> >> > echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
>> >> > echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
>> >> > echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind
>> >> >
>> >> > I'll try it again with the fourth line added, manually before I start the VM.
>> >> >
>> >> > How come yours is 'echo "PCI" > /sys/bus/drivers/DRIVERNAME/unbind'
>> >> > and mine is echo "PCI" > /sys/bus/pci/devices/PCI/driver/unbind
>> >> >
>> >> > Obviously one looks up which driver is being used by the PCI id, but
>> >> > how do I look up which driver my PCI card is using?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > James.
>> >> >
>> >>
>> >> Hi,
>> >>
>> >> OK, adding the fourth line does nothing, changing the second line to
>> >> "driver" rather than "device" does nothing, including in combination
>> >> with fourth line there/not there.
>> >
>> > Yep, the last line is just removing the id from pci-stub, it doesn't
>> > change anything about devices that are already bound to it.  driver vs
>> > device are just different ways to get to the same thing.
>> >
>> >> So I'm still stuck, can anybody else help?
>> >> Perhaps point me to a guide on how to compile the latest qemu-kvm
>> >> against my kernel?
>> >
>> > I don't know why you're getting -EBUSY for this device, but maybe we can
>> > start from a clean slate and see if it helps.  Here's what I would
>> > suggest:
>> >
>> > echo "0000:08:06.0" > /sys/bus/pci/devices/0000:08:06.0/driver/unbind
>> > echo "0000:08:06.1" > /sys/bus/pci/devices/0000:08:06.1/driver/unbind
>> > echo "0000:08:06.2" > /sys/bus/pci/devices/0000:08:06.2/driver/unbind
>> > echo "0000:08:0e.0" > /sys/bus/pci/devices/0000:08:0e.0/driver/unbind
>> >
>> > Note we have to knock out the firewire because it shares an interrupt
>> > with the ehci device you're trying to assign.  We want to remove the USB
>> > controller entirely from the host.  Your dmesg indicates the host is
>> > still seeing the device via the uhci ports, and isn't happy about it.
>> > You can ignore pci-stub for the moment, it's just a way to keep drivers
>> > from claiming the device, it's not required for device assignment.  Now,
>> > instead of only trying to assign the ehci, let's move the whole usb
>> > controller to the guest:
>> >
>> > -device pci-assign,host=08:06.0,addr=5.0 \
>> > -device pci-assign,host=08:06.1,addr=5.1 \
>> > -device pci-assign,host=08:06.2,addr=5.2
>> >
>> > (slot 5 on the guest is arbitrary, pick something else if you need to)
>> > If that works, then you can bind all those devices to pci-stub and it
>> > should still work.
>> >
>> > Alex
>>
>> Hi,
>>
>> Sorry about the slow reply, I hosed all of my PCs fiddling with
>> compiling the latest qemu, took me a while to put it all back together
>> again in between work!
>>
>> I'm afraid I use virtmanager, although I guess using the "add pci
>> device" function is the same as -device pci-assign? It does seem to
>> add it to the command line that gets written out to the log files in
>> /var/log/libvirt/qemu.
>>
>> Nevertheless, I've tried that and still not luck, the log output is
>> similar, with one extra line:
>>
>> http://pastebin.com/MJ6aqjNq
>
> You actually ended up with:
>
> -device pci-assign,host=08:06.0,id=hostdev0,configfd=58,bus=pci.0,addr=0x6 \
> -device pci-assign,host=08:06.1,id=hostdev1,configfd=59,bus=pci.0,addr=0x7 \
> -device pci-assign,host=08:06.2,id=hostdev2,configfd=60,bus=pci.0,addr=0x8
>
> Which isn't quite what I was suggesting.  You probably have xml that
> looks like this:
>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>      </source>
>    </hostdev>
>    ...
>
> Try adding an address line, so you get something more like this:
>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
>    </hostdev>
>    <hostdev mode='subsystem' type='pci' managed='yes'>
>      <source>
>        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
>      </source>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
>    </hostdev>
>
>
>> Using raw in/out ioport access (sysfs - Input/output error)
>
> This is just an informational line to let us know whether pci-sysfs
> supports read/write on the ioport resource files.  If it does, we use
> those rather than doing raw in/out for access to the device.  This does
> highlight another potential problem.  Your distro probably doesn't have
> all the patches in place for non-privileged device assignment, which
> could be why you're having strange issues.  Check
> your /etc/libvirt/qemu.conf for the 'user =' and 'group =' lines.  If
> they're not already, try setting them to 'root', restart libvirtd and
> see if anything improves.
>
>> Here's the dmesg output:
>> http://pastebin.com/AE1euUN1
>
> If still issues after the above, it might be useful to pastbin the
> entire dmesg so we can make sure the iommu is really on.  Thanks,
>
> Alex

Hi,

No such luck I'm afraid.

Here is the original XML:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
    </hostdev>

The only difference from your recommended change is the target (I take
it they are target addresses for the VM?) addresses run:
0000:00:06.0
0000:00:07.0
0000:00:08.0

Instead of:
0000:00:06.0
0000:00:06.1
0000:00:06.2

Regardless, I still changed test.xml to:
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x06' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x2'/>
    </hostdev>

To no effect.

In qemu.conf, user and group were commented out, I uncommented both
and they were both already set to root.

After both a restart of libvirt-bin and the pc itself, no change.

Finally, here is the very latest dmesg:
http://pastebin.com/9HE61K62

Does anybody know the debug kernel switches for iommu?

Many Thanks,

James.