Re: [RFC] ARM PCI Passthrough design document

[Julien Grall <julien.grall@linaro.org>]

On 05/29/2017 03:30 AM, Manish Jaggi wrote:
> Hi Julien,

Hello Manish,

> On 5/26/2017 10:44 PM, Julien Grall wrote:
>> PCI pass-through allows the guest to receive full control of physical PCI
>> devices. This means the guest will have full and direct access to the PCI
>> device.
>>
>> ARM is supporting a kind of guest that exploits as much as possible
>> virtualization support in hardware. The guest will rely on PV driver only
>> for IO (e.g block, network) and interrupts will come through the
>> virtualized
>> interrupt controller, therefore there are no big changes required
>> within the
>> kernel.
>>
>> As a consequence, it would be possible to replace PV drivers by
>> assigning real
>> devices to the guest for I/O access. Xen on ARM would therefore be
>> able to
>> run unmodified operating system.
>>
>> To achieve this goal, it looks more sensible to go towards emulating the
>> host bridge (there will be more details later).
> IIUC this means that domU would have an emulated host bridge and dom0
> will see the actual host bridge?

You don't want the hardware domain and Xen access the configuration 
space at the same time. So if Xen is in charge of the host bridge, then 
an emulated host bridge should be exposed to the hardware.

Although, this is depending on who is in charge of the the host bridge. 
As you may have noticed, this design document is proposing two ways to 
handle configuration space access. At the moment any generic host bridge 
(see the definition in the design document) will be handled in Xen and 
the hardware domain will have an emulated host bridge.

If your host bridges is not a generic one, then the hardware domain will 
be  in charge of the host bridges, any configuration access from Xen 
will be forward to the hardware domain.

At the moment, as part of the first implementation, we are only looking 
to implement a generic host bridge in Xen. We will decide on case by 
case basis for all the other host bridges whether we want to have the 
driver in Xen.

[...]

>> ## IOMMU
>>
>> The IOMMU will be used to isolate the PCI device when accessing the
>> memory (e.g
>> DMA and MSI Doorbells). Often the IOMMU will be configured using a
>> MasterID
>> (aka StreamID for ARM SMMU)  that can be deduced from the SBDF with
>> the help
>> of the firmware tables (see below).
>>
>> Whilst in theory, all the memory transactions issued by a PCI device
>> should
>> go through the IOMMU, on certain platforms some of the memory
>> transaction may
>> not reach the IOMMU because they are interpreted by the host bridge. For
>> instance, this could happen if the MSI doorbell is built into the PCI
>> host
>> bridge or for P2P traffic. See [6] for more details.
>>
>> XXX: I think this could be solved by using direct mapping (e.g GFN ==
>> MFN),
>> this would mean the guest memory layout would be similar to the host
>> one when
>> PCI devices will be pass-throughed => Detail it.
> In the example given in the IORT spec, for pci devices not behind an SMMU,
> how would the writes from the device be protected.

I realize the XXX paragraph is quite confusing. I am not trying to solve 
the problem where PCI devices are not protected behind an SMMU but 
platform where some transactions (e.g P2P or MSI doorbell access) are 
by-passing the SMMU.

You may still want to allow PCI passthrough in that case, because you 
know that P2P cannot be done (or potentially disabled) and MSI doorbell 
access is protected (for instance a write in the ITS doorbell will be 
tagged with the device by the hardware). In order to support such 
platform you need to direct map the doorbel (e.g GFN == MFN) and carve 
out the P2P region from the guest memory map. Hence the suggestion to 
re-use the host memory layout for the guest.

Note that it does not mean the RAM region will be direct mapped. It is 
only there to ease carving out memory region by-passed by the SMMU.

[...]

>> ## ACPI
>>
>> ### Host bridges
>>
>> The static table MCFG (see 4.2 in [1]) will describe the host bridges
>> available
>> at boot and supporting ECAM. Unfortunately, there are platforms out there
>> (see [2]) that re-use MCFG to describe host bridge that are not fully
>> ECAM
>> compatible.
>>
>> This means that Xen needs to account for possible quirks in the host
>> bridge.
>> The Linux community are working on a patch series for this, see [2]
>> and [3],
>> where quirks will be detected with:
>>      * OEM ID
>>      * OEM Table ID
>>      * OEM Revision
>>      * PCI Segment
>>      * PCI bus number range (wildcard allowed)
>>
>> Based on what Linux is currently doing, there are two kind of quirks:
>>      * Accesses to the configuration space of certain sizes are not
>> allowed
>>      * A specific driver is necessary for driving the host bridge
>>
>> The former is straightforward to solve but the latter will require
>> more thought.
>> Instantiation of a specific driver for the host controller can be
>> easily done
>> if Xen has the information to detect it.
> So Xen would parse the MCFG to find a hb, then map the config space in
> dom0 stage2 ?
> and then provide the same MCFG to dom0?

This is implementation details. I have been really careful so far to 
leave the implementation open as it does not matter at this stage how we 
are going to implement it in Xen.

[...]

>> ## Discovering and registering host bridge
>>
>> The approach taken in the document will require communication between
>> Xen and
>> the hardware domain. In this case, they would need to agree on the
>> segment
>> number associated to an host bridge. However, this number is not
>> available in
>> the Device Tree case.
>>
>> The hardware domain will register new host bridges using the existing
>> hypercall
>> PHYSDEV_mmcfg_reserved:
>>
>> #define XEN_PCI_MMCFG_RESERVED 1
>>
>> struct physdev_pci_mmcfg_reserved {
>>      /* IN */
>>      uint64_t    address;
>>      uint16_t    segment;
>>      /* Range of bus supported by the host bridge */
>>      uint8_t     start_bus;
>>      uint8_t     end_bus;
>>
>>      uint32_t    flags;
>> }
> So this hypercall is not required for ACPI?

This is not DT specific as even on ACPI there are platform not fully 
ECAM compliant. As I said above, we will need to decide whether we want 
to support non-ECAM compliant host bridges (e.g all host bridges have a 
specific drivers) in Xen. Likely this will be on case by case basis.

[...]

>> ## Discovering and registering PCI devices
>>
>> The hardware domain will scan the host bridge to find the list of PCI
>> devices
>> available and then report it to Xen using the existing hypercall
>> PHYSDEV_pci_device_add:
>>
>> #define XEN_PCI_DEV_EXTFN   0x1
>> #define XEN_PCI_DEV_VIRTFN  0x2
>> #define XEN_PCI_DEV_PXM     0x3
>>
>> struct physdev_pci_device_add {
>>      /* IN */
>>      uint16_t    seg;
>>      uint8_t     bus;
>>      uint8_t     devfn;
>>      uint32_t    flags;
>>      struct {
>>          uint8_t bus;
>>          uint8_t devfn;
>>      } physfn;
>>      /*
>>       * Optional parameters array.
>>       * First element ([0]) is PXM domain associated with the device (if
>>       * XEN_PCI_DEV_PXM is set)
>>       */
>>      uint32_t optarr[0];
>> }
> For mapping the MMIO space of the device in Stage2, we need to add
> support in Xen / via a map hypercall in linux/drivers/xen/pci.c

Mapping MMIO space in stage-2 is not PCI specific and already addressed 
in Xen 4.9 (see commit 80f9c31 "xen/arm: acpi: Map MMIO on fault in 
stage-2 page table for the hardware domain"). So I don't understand why 
we should care about that here...

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel