Device assignment, shared IRQs, uio_pci_generic

Device assignment, shared IRQs, uio_pci_generic

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Hi all,

trying to pass some host device to a KVM guest (for driver hacking), I
ran into massive IRQ sharing issues on the host. Classic device
assignment refuses to work in this case. I remembered uio_pci_generic,
which I thought was once designed to resolve this conflict. But it isn't
used by KVM or qemu, is it?

So I crossed fingers and hacked IRQF_SHARED into
assigned_device_enable_host_intx. Effect as expected: assignment work,
host gets massive hick-ups (the graphic adapter is another user of this
line...).

Are there already VFIO patches ready for testing that work with shared
IRQs as well? And/or would it make sense to integrate IRQ sharing via
PCI-level IRQ masking to the KVM core until VFIO becomes standard?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Device assignment, shared IRQs, uio_pci_generic

[Alex Williamson]

On Sat, 2010-10-30 at 17:58 +0200, Jan Kiszka wrote:
> Hi all,
> 
> trying to pass some host device to a KVM guest (for driver hacking), I
> ran into massive IRQ sharing issues on the host. Classic device
> assignment refuses to work in this case. I remembered uio_pci_generic,
> which I thought was once designed to resolve this conflict. But it isn't
> used by KVM or qemu, is it?

Nope, VFIO started out being based on it, but has since branched off.
KVM has never made use of it.

> So I crossed fingers and hacked IRQF_SHARED into
> assigned_device_enable_host_intx. Effect as expected: assignment work,
> host gets massive hick-ups (the graphic adapter is another user of this
> line...).
> 
> Are there already VFIO patches ready for testing that work with shared
> IRQs as well? And/or would it make sense to integrate IRQ sharing via
> PCI-level IRQ masking to the KVM core until VFIO becomes standard?

There's a very old qemu VFIO patch floating around, but I plan to send
out a new one this week, thus all the pre-patches I've been sending out.
Adding PCI 2.3 interrupt disable support to KVM should be very possible,
and I suspect fairly easy.  Oddly, I've just been adding support for
non-PCI 2.3 devices to VFIO so it can support all the old devices that
KVM does already.

Alex

Re: Device assignment, shared IRQs, uio_pci_generic

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

Am 30.10.2010 19:06, Alex Williamson wrote:
> On Sat, 2010-10-30 at 17:58 +0200, Jan Kiszka wrote:
>> Hi all,
>>
>> trying to pass some host device to a KVM guest (for driver hacking), I
>> ran into massive IRQ sharing issues on the host. Classic device
>> assignment refuses to work in this case. I remembered uio_pci_generic,
>> which I thought was once designed to resolve this conflict. But it isn't
>> used by KVM or qemu, is it?
> 
> Nope, VFIO started out being based on it, but has since branched off.
> KVM has never made use of it.
> 
>> So I crossed fingers and hacked IRQF_SHARED into
>> assigned_device_enable_host_intx. Effect as expected: assignment work,
>> host gets massive hick-ups (the graphic adapter is another user of this
>> line...).
>>
>> Are there already VFIO patches ready for testing that work with shared
>> IRQs as well? And/or would it make sense to integrate IRQ sharing via
>> PCI-level IRQ masking to the KVM core until VFIO becomes standard?
> 
> There's a very old qemu VFIO patch floating around, but I plan to send
> out a new one this week, thus all the pre-patches I've been sending out.
> Adding PCI 2.3 interrupt disable support to KVM should be very possible,
> and I suspect fairly easy.  Oddly, I've just been adding support for
> non-PCI 2.3 devices to VFIO so it can support all the old devices that
> KVM does already.

It wasn't that easy, and I'm still not 100% sure I did it all correctly:
I just passed my SD card reader to a Linux guest which shares IRQ with
five(!) other devices. That works well now (provided I disable SMIs -
device pass-through appears to be another trigger for my SMI-related
lockups under KVM, sigh).

Still not working here are ehci, ath9k and e1000e when passed through.
They receive IRQs, but somehow the Linux guest drivers are unhappy about
the device states (the e1000e detects "Hardware Unit Hang" e.g.). This
is independent of my patches. Anyone any ideas? Improper configurations
of the BIOS? I stumbled over this in the kernel log:

    DMAR: BIOS has allocated no shadow GTT; disabling IOMMU for graphics

There were a few flaws in the current device assignment code /wrt host
IRQ type selection, both kernel and qemu-kvm are affected. Will try to
sort all my stuff next to send out patches.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: Device assignment, shared IRQs, uio_pci_generic

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 966 bytes --]

Am 31.10.2010 16:05, Jan Kiszka wrote:
> Still not working here are ehci, ath9k and e1000e when passed through.
> They receive IRQs, but somehow the Linux guest drivers are unhappy about
> the device states (the e1000e detects "Hardware Unit Hang" e.g.). This
> is independent of my patches. Anyone any ideas? Improper configurations
> of the BIOS? I stumbled over this in the kernel log:
> 
>     DMAR: BIOS has allocated no shadow GTT; disabling IOMMU for graphics
> 

Argh, no one told me that there is CONFIG_DMAR_DEFAULT_ON and that you
have to pass "intel_iommu=on" if that config switch is off like here.
Moreover, I somehow assumed "iommu=0" on the command line means "use
IOMMU with index 0" (don't ask me why). Usability deserves some patches
- later.

So it's starting to work, already accessed some devices (ath9k did a
successful scan), but now it's crashing in intel_iommu_attach_device.
Looks like use after release. Sigh.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]