Re: [early RFC] ARM PCI Passthrough design document

[Julien Grall <julien.grall@linaro.org>]

Hi Stefano,

On 04/01/17 00:24, Stefano Stabellini wrote:
> On Thu, 29 Dec 2016, Julien Grall wrote:

[...]

>> # Introduction
>>
>> PCI passthrough allows to give control of physical PCI devices to guest. This
>> means that the guest will have full and direct access to the PCI device.
>>
>> ARM is supporting one kind of guest that is exploiting as much as possible
>> virtualization support in hardware. The guest will rely on PV driver only
>> for IO (e.g block, network), interrupts will come through the virtualized
>> interrupt controller. This means that there are no big changes required
>> within the kernel.
>>
>> By consequence, it would be possible to replace PV drivers by assigning real
>   ^ As a consequence

I will fix all the typoes in the next version.

>
>
>> devices to the guest for I/O access. Xen on ARM would therefore be able to
>> run unmodified operating system.

[...]

>> Instantiation of a specific driver for the host controller can be easily done
>> if Xen has the information to detect it. However, those drivers may require
>> resources described in ASL (see [4] for instance).
>>
>> XXX: Need more investigation to know whether the missing information should
>> be passed by DOM0 or hardcoded in the driver.
>
> Given that we are talking about quirks here, it would be better to just
> hardcode them in the drivers, if possible.

Indeed hardcoded would be the preferred way to avoid introduce new 
hypercall for quirk.

For instance, in the case of Thunder-X (see commit 44f22bd "PCI: Add 
MCFG quirks for Cavium ThunderX pass2.x host controller) some region are 
read from ACPI. What I'd like to understand is whether this could be 
hardcoded or can it change between platform? If it can change, is there 
a way in ACPI to differentiate 2 platforms?

Maybe this is a question that Cavium can answer? (in CC).


[...]

>> ## Discovering and register hostbridge
>>
>> Both ACPI and Device Tree do not provide enough information to fully
>> instantiate an host bridge driver. In the case of ACPI, some data may come
>> from ASL,
>
> The data available from ASL is just to initialize quirks and non-ECAM
> controllers, right? Given that SBSA mandates ECAM, and we assume that
> ACPI is mostly (if not only) for servers, then I think it is safe to say
> that in the case of ACPI we should have all the info to fully
> instantiate an host bridge driver.

 From the spec, the MCFG will only describe host bridge available at 
boot (see 4.2 in "PCI firmware specification, rev 3.2"). All the other 
host bridges will be described in ASL.

So we need DOM0 to feed Xen about the latter host bridges.

>
>
>> whilst for Device Tree the segment number is not available.
>>
>> So Xen needs to rely on DOM0 to discover the host bridges and notify Xen
>> with all the relevant informations. This will be done via a new hypercall
>> PHYSDEVOP_pci_host_bridge_add. The layout of the structure will be:
>
> I understand that the main purpose of this hypercall is to get Xen and Dom0 to
> agree on the segment numbers, but why is it necessary? If Dom0 has an
> emulated contoller like any other guest, do we care what segment numbers
> Dom0 will use?

I was not planning to have a emulated controller for DOM0. The physical 
one is not necessarily ECAM compliant so we would have to either emulate 
the physical one (meaning multiple different emulation) or an ECAM 
compliant.

The latter is not possible because you don't know if there is enough 
free MMIO space for the emulation.

In the case on ARM, I don't see much the point to emulate the host 
bridge for DOM0. The only thing we need in Xen is to access the 
configuration space, we don't have about driving the host bridge. So I 
would let DOM0 dealing with that.

Also, I don't see any reason for ARM to trap DOM0 configuration space 
access. The MSI will be configured using the interrupt controller and it 
is a trusted Domain.

>
>
>> struct physdev_pci_host_bridge_add
>> {
>>     /* IN */
>>     uint16_t seg;
>>     /* Range of bus supported by the host bridge */
>>     uint8_t  bus_start;
>>     uint8_t  bus_nr;
>>     uint32_t res0;  /* Padding */
>>     /* Information about the configuration space region */
>>     uint64_t cfg_base;
>>     uint64_t cfg_size;
>> }
>>
>> DOM0 will issue the hypercall PHYSDEVOP_pci_host_bridge_add for each host
>> bridge available on the platform. When Xen is receiving the hypercall, the
>> the driver associated to the host bridge will be instantiated.
>
> I think we should mention the relationship with the existing
> PHYSDEVOP_pci_mmcfg_reserved hypercall.

Sorry, I did not spot this hypercall until now. From a brief look, this 
hypercall would be redundant, but I will investigate a bit more.

>
>
>> XXX: Shall we limit DOM0 the access to the configuration space from that
>> moment?
>
> If we can, we should

Why would be the benefits? For now, I see a big drawback: resetting a 
PCI devices would need to be done in Xen rather than DOM0. As you may 
now there are a lot of quirks for reset.

So for me, it looks more sensible to handle this in DOM0 and let DOM0 a 
full access to the configuration space. Overall he is a trusted domain.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel