[Qemu-devel] vfio with iommu groups

[Qemu-devel] vfio with iommu groups

[Alexey Kardashevskiy]

Hi!

I pulled new VFIO from github, ported to POWER and got some issues/thoughts which I post as patches.
However PCI bridges handling is an open question to discuss.

My test setup includes PCIe card Intel E1000E which looks in the host like this
(cut device names in the tree below as they were too long to look nice):
aik@vpl2:~$ lspci -vt
 +-[0003:00]---00.0-[01-ff]----00.0-[02-04]--+-02.0-[03]--+-00.0  Intel Gigabit Ethernet
 |                                           |            \-00.1  Intel Gigabit Ethernet
 |                                           \-04.0-[04]--+-00.0  Intel Gigabit Ethernet
 |                                                        \-00.1  Intel Gigabit Ethernet

aik@vpl2:~$ lspci
...
0003:00:00.0 PCI bridge: IBM Device 02ea (rev 02)
0003:01:00.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e)
0003:02:02.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e)
0003:02:04.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e)
0003:03:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
(rev 06)
0003:03:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
(rev 06)
0003:04:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
(rev 06)
0003:04:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
(rev 06)


At the moment I bind all devices from domain 0003 to the vfio_pci driver and give as many
devices from it to QEMU as I want. vfio-pci owns devices, the guest sees what he needs to see -
ethernet adapters, it works.

However theoretically we might want to show these 3 PCIe bridges as well (but not the root complex).
For example, INTx lines should be swizzled when the guest parses a device tree and
tries to calculate a real IRQ number. VFIO does not handles bridges at all, it treats them as PCI
functions.

Is there any idea what to do with bridges?


-- 
Alexey

Re: [Qemu-devel] vfio with iommu groups

[Alex Williamson]

On Sat, 2012-05-12 at 17:31 +1000, Alexey Kardashevskiy wrote:
> Hi!
> 
> I pulled new VFIO from github, ported to POWER and got some issues/thoughts which I post as patches.
> However PCI bridges handling is an open question to discuss.
> 
> My test setup includes PCIe card Intel E1000E which looks in the host like this
> (cut device names in the tree below as they were too long to look nice):
> aik@vpl2:~$ lspci -vt
>  +-[0003:00]---00.0-[01-ff]----00.0-[02-04]--+-02.0-[03]--+-00.0  Intel Gigabit Ethernet
>  |                                           |            \-00.1  Intel Gigabit Ethernet
>  |                                           \-04.0-[04]--+-00.0  Intel Gigabit Ethernet
>  |                                                        \-00.1  Intel Gigabit Ethernet
> 
> aik@vpl2:~$ lspci
> ...
> 0003:00:00.0 PCI bridge: IBM Device 02ea (rev 02)
> 0003:01:00.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e)
> 0003:02:02.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e)
> 0003:02:04.0 PCI bridge: Integrated Device Technology, Inc. PES12N3A PCI Express Switch (rev 0e)
> 0003:03:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
> (rev 06)
> 0003:03:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
> (rev 06)
> 0003:04:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
> (rev 06)
> 0003:04:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper)
> (rev 06)
> 
> 
> At the moment I bind all devices from domain 0003 to the vfio_pci driver and give as many
> devices from it to QEMU as I want. vfio-pci owns devices, the guest sees what he needs to see -
> ethernet adapters, it works.

Yay!

> However theoretically we might want to show these 3 PCIe bridges as well (but not the root complex).
> For example, INTx lines should be swizzled when the guest parses a device tree and
> tries to calculate a real IRQ number. VFIO does not handles bridges at all, it treats them as PCI
> functions.
> 
> Is there any idea what to do with bridges?

I don't really have a problem with the idea of exposing bridges, but it
get's pretty complicated.  What happens if the guest reprograms the
subordinate bus numbers?  Is there any advantage vs exposing an emulated
bridge in it's place?  Do you have a proposal for it?  Thanks,

Alex

Re: [Qemu-devel] vfio with iommu groups

[Benjamin Herrenschmidt]

On Sun, 2012-05-13 at 22:16 -0600, Alex Williamson wrote:
> > However theoretically we might want to show these 3 PCIe bridges as well (but not the root complex).
> > For example, INTx lines should be swizzled when the guest parses a device tree and
> > tries to calculate a real IRQ number. VFIO does not handles bridges at all, it treats them as PCI
> > functions.
> > 
> > Is there any idea what to do with bridges?
> 
> I don't really have a problem with the idea of exposing bridges, but it
> get's pretty complicated.  What happens if the guest reprograms the
> subordinate bus numbers?  Is there any advantage vs exposing an emulated
> bridge in it's place?  Do you have a proposal for it?  Thanks,

I think exposing an emulated bridge would be more in the "spirit" of the
current vfio and safer for now (long run, as we already discussed, we
might look at an other option with more direct HW access for power).

We do want to expose those bridges one way or another for the sake of
getting the interrupt routing right, since the guest will make
assumptions based on those bridges to calculate the proper swizzling.

Cheers,
Ben.