Re: PCI Pass-through in Xen ARM: Draft 4

[David Daney <ddaney@caviumnetworks.com>]

On 09/15/2015 11:58 AM, Jaggi, Manish wrote:
> 
> 
> On Thursday 10 September 2015 06:42 AM, Julien Grall wrote:
>> Hi Manish,
> Hi Julien, sorry for late response..
>>
>> On 13/08/2015 10:42, Manish Jaggi wrote:
>>>    3.2.    Mapping between streamID - deviceID - pci sbdf - requesterID
>>> -----------------------------------------------------------------------------
>>>    For a simpler case all should be equal to BDF. But there are some
>>> devices
>>>    that use the wrong requester ID for DMA transactions. Linux kernel has
>>> PCI
>>>    quirks for these. How the same be implemented in Xen or a diffrent
>>> approach
>>>    has to be taken is TODO here.
>>>
>>>    Till that time, for basic implementation it is assumed that all are
>>> equal
>>>    to BDF.
>>
>> Back to this streamID = DeviceID = requestID = SBDF again...
>>
>> I've just found a patch for Linux send by one of your colleague about
>> tweaking the requesterID for thunder-X board (See [1]). The most
>> interesting bits are:
>>
>> static u32 pci_requester_id_ecam(struct pci_dev *dev)
>> {
>>      return (((pci_domain_nr(dev->bus) >> 2) << 19) |
>>          ((pci_domain_nr(dev->bus) % 4) << 16) |
>>          (dev->bus->number << 8) | dev->devfn);
>> }
>>
>> static u32 thunder_pci_requester_id(struct pci_dev *dev, u16 alias)
>> {
>>      int ret;
>>
>>      ret = thunder_pem_requester_id(dev);
>>      if (ret >= 0)
>>          return (u32)ret;
>>
>>      return pci_requester_id_ecam(dev);
>> }
>>
>> Which is then used to override the default function used by ITS to
>> find the deviceID.
>>
>> AFAICT, this means that you can't safely assume that DeviceID == sBDF
>> even for your platform. Is that right?
> Yes.
>>
>> If so, I'm afraid that you have to handle DeviceID != sBDF (and so on)
>> in your design doc. I.e how do you plan to get the base requester ID.
>>
> Right. But on our platform the requesterId is uniquely generated from
> bdf. Adding David to confirm that.

The requesterId is: Node << 19 + ECAM# << 16 + bdf

For the Linux kernel, the Node/ECAM part is going to be read from OF
device-tree or ACPI, not calculated with the above formula.


>> I can see 2 different solutions:
>>      1) Let DOM0 pass the first requester ID when registering the bus
>>         Pros:
>>          * Less per-platform code in Xen
>>         Cons:
>>          * Assume that the requester ID are contiguous. (Is it really a
>> cons?)
>>          * Still require quirk for buggy device (i.e requester ID not
>> correct)
>>      2) Do it in Xen
>>         Pros:
>>          * We are not relying on DOM0 giving the requester ID
>>              => Not assuming contiguous requester ID
>>          Cons:
>>          * Per PCI bridge code to handle the mapping
>>
>    We can have (3) that when PHYSDEVOP_pci_add_device is called the sbdf
> and requesterID both are passed in hypercall.
>> Regards,
>>
>> [1] https://lkml.org/lkml/2015/7/15/703
>>
>>
>>
>