Re: PCI devices passthrough on Arm design proposal

[Bertrand Marquis <Bertrand.Marquis@arm.com>]

> On 17 Jul 2020, at 18:05, Roger Pau Monné <roger.pau@citrix.com> wrote:
> 
> On Fri, Jul 17, 2020 at 03:47:25PM +0000, Bertrand Marquis wrote:
>>> On 17 Jul 2020, at 17:26, Julien Grall <julien@xen.org> wrote:
>>> On 17/07/2020 15:47, Bertrand Marquis wrote:
>>>>>>> * Dom0Less implementation will require to have the capacity inside Xen to discover the PCI devices (without depending on Dom0 to declare them to Xen).
>>>>>>> 
>>>>>>> # Enable the existing x86 virtual PCI support for ARM:
>>>>>>> 
>>>>>>> The existing VPCI support available for X86 is adapted for Arm. When the device is added to XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config space access is added to the PCI device to emulate the PCI devices.
>>>>>>> 
>>>>>>> A MMIO trap handler for the PCI ECAM space is registered in XEN so that when guest is trying to access the PCI config space, XEN will trap the access and emulate read/write using the VPCI and not the real PCI hardware.
>>>>>>> 
>>>>>>> Limitation:
>>>>>>> * No handler is register for the MSI configuration.
>>>>>>> * Only legacy interrupt is supported and tested as of now, MSI is not implemented and tested.
>>>>>> IIRC, legacy interrupt may be shared between two PCI devices. How do you plan to handle this on Arm?
>>>> We plan to fix this by adding proper support for MSI in the long term.
>>>> For the use case where MSI is not supported or not wanted we might have to find a way to forward the hardware interrupt to several guests to emulate some kind of shared interrupt.
>>> 
>>> Sharing interrupts are a bit pain because you couldn't take advantage of the direct EOI in HW and have to be careful if one guest doesn't EOI in timely maneer.
>>> 
>>> This is something I would rather avoid unless there is a real use case for it.
>> 
>> I would expect that most recent hardware will support MSI and this
>> will not be needed.
> 
> Well, PCI Express mandates MSI support, so while this is just a spec,
> I would expect most (if not all) devices to support MSI (or MSI-X), as
> Arm platforms haven't implemented legacy PCI anyway.

Yes that’s our assumption to. But we have to start somewhere so MSI is
planned but in a future step. I would think that supporting non MSI if not
impossible will be a lot more complex due to the interrupt sharing.
I do think that not supporting non MSI should be ok on Arm.

> 
>> When MSI is not used, the only solution would be to enforce that
>> devices assigned to different guest are using different interrupts
>> which would limit the number of domains being able to use PCI
>> devices on a bus to 4 (if the enumeration can be modified correctly
>> to assign the interrupts properly).
>> 
>> If we all agree that this is an acceptable limitation then we would
>> not need the “interrupt sharing”.
> 
> I might be easier to start by just supporting devices that have MSI
> (or MSI-X) and then move to legacy interrupts if required?

MSI support requires also some support in the interrupt controller part
on arm. So there is some work to achieve that.

> 
> You should have most of the pieces you require already implemented
> since that's what x86 uses, and hence could reuse almost all of it?

Inside PCI probably but the GIC part will require some work.

> 
> IIRC Julien even said that Arm was likely to require much less traps
> than x86 for accesses to MSI and MSI-X since you could allow untrusted
> guests to write directly to the registers as there's another piece of
> hardware that would already translate the interrupts?

Yes this is definitely the case. The ITS part of the GIC interrupt controller
will help a lot and reduce the number of traps.

> 
> I think it's fine to use this workaround while you don't have MSI
> support in order to start testing and upstreaming stuff, but maybe
> that shouldn't be committed?

That was definitely not our plan to commit the code without MSI.
But as requested during the Xen Summit, we try to publish some code
for an RFC and a design early to get comment from the community and
we try to do that with something working, even partially and with lots of
limitations.

Bertrand