Re: PCI Pass-through in Xen ARM - Draft 2.

[Manish Jaggi <mjaggi@caviumnetworks.com>]

On Thursday 30 July 2015 03:24 PM, Ian Campbell wrote:
> On Wed, 2015-07-29 at 15:07 +0530, Manish Jaggi wrote:
>> On Monday 06 July 2015 03:50 PM, Ian Campbell wrote:
>>> On Mon, 2015-07-06 at 15:36 +0530, Manish Jaggi wrote:
>>>> On Monday 06 July 2015 02:41 PM, Ian Campbell wrote:
>>>>> On Sun, 2015-07-05 at 11:25 +0530, Manish Jaggi wrote:
>>>>>> On Monday 29 June 2015 04:01 PM, Julien Grall wrote:
>>>>>>> Hi Manish,
>>>>>>>
>>>>>>> On 28/06/15 19:38, Manish Jaggi wrote:
>>>>>>>> 4.1 Holes in guest memory space
>>>>>>>> ----------------------------
>>>>>>>> Holes are added in the guest memory space for mapping pci
>>>>>>>> device's BAR
>>>>>>>> regions.
>>>>>>>> These are defined in arch-arm.h
>>>>>>>>
>>>>>>>> /* For 32bit */
>>>>>>>> GUEST_MMIO_HOLE0_BASE, GUEST_MMIO_HOLE0_SIZE
>>>>>>>>      
>>>>>>>> /* For 64bit */
>>>>>>>> GUEST_MMIO_HOLE1_BASE , GUEST_MMIO_HOLE1_SIZE
>>>>>>> The memory layout for 32bit and 64bit are exactly the same. Why
>>>>>>> do you
>>>>>>> need to differ here?
>>>>>> I think Ian has already replied. I will change the name of macro
>>>>>>>> 4.2 New entries in xenstore for device BARs
>>>>>>>> --------------------------------------------
>>>>>>>> toolkit also updates the xenstore information for the device
>>>>>>>> (virtualbar:physical bar).
>>>>>>>> This information is read by xenpciback and returned to the
>>>>>>>> pcifront
>>>>>>>> driver configuration
>>>>>>>> space accesses.
>>>>>>> Can you details what do you plan to put in xenstore and how?
>>>>>> It is implementation . But I plan to put under domU / device /
>>>>>> heirarchy
>>>>> Actually, xenstore is an API of sorts which needs to be maintained
>>>>> going
>>>>> forward (since front and backend can evolve separately, so it does
>>>>> need
>>>>> some level of design and documentation.
>>>>>
>>>>>>> What about the expansion ROM?
>>>>>> Do you want to put some restriction on not using expansion ROM as
>>>>>> a
>>>>>> passthrough device.
>>>>> "expansion ROM as a passthrough device" doesn't make sense to me,
>>>>> passthrough devices may _have_ an expansion ROM.
>>>>>
>>>>> The expansion ROM is just another BAR. I don't know how
>>>>> pcifront/back
>>>>> deal with those today on PV x86, but I see no reason for ARM to
>>>>> deviate.
>>>>>
>>>>>
>>>>>>>> 4.3 Hypercall for bdf mapping notification to xen
>>>>>>>> -----------------------------------------------
>>>>>>>> #define PHYSDEVOP_map_sbdf              43
>>>>>>>> typedef struct {
>>>>>>>>         u32 s;
>>>>>>>>         u8 b;
>>>>>>>>         u8 df;
>>>>>>>>         u16 res;
>>>>>>>> } sbdf_t;
>>>>>>>> struct physdev_map_sbdf {
>>>>>>>>         int domain_id;
>>>>>>>>         sbdf_t    sbdf;
>>>>>>>>         sbdf_t    gsbdf;
>>>>>>>> };
>>>>>>>>
>>>>>>>> Each domain has a pdev list, which contains the list of all
>>>>>>>> pci devices.
>>>>>>>> The
>>>>>>>> pdev structure already has a sbdf information. The
>>>>>>>> arch_pci_dev is
>>>>>>>> updated to
>>>>>>>> contain the gsbdf information. (gs- guest segment id)
>>>>>>>>
>>>>>>>> Whenever there is trap from guest or an interrupt has to be
>>>>>>>> injected,
>>>>>>>> the pdev
>>>>>>>> list is iterated to find the gsbdf.
>>>>>>> Can you give more background for this section? i.e:
>>>>>>> 	- Why do you need this?
>>>>>>> 	- How xen will translate the gbdf to a vDeviceID?
>>>>>> In the context of the hypercall processing.
>>>>>>> 	- Who will call this hypercall?
>>>>>>> 	- Why not setting the gsbdf when the device is
>>>>>>> assigned?
>>>>>> Can the maintainer of the pciback suggest an alternate.
>>>>> That's not me, but I don't think this belongs here, I think it can
>>>>> be
>>>>> done from the toolstack. If you think not then please explain what
>>>>> information the toolstack doesn't have in its possession which
>>>>> prevents
>>>>> this mapping from being done there.
>>>> The toolstack does not have the guest sbdf information. I could only
>>>> find it in xenpciback.
>>> Are you sure? The sbdf relates to the physical device, correct? If so
>>> then surely the toolstack knows it -- it's written in the config file
>>> and is the primary parameter to all of the related libxl passthrough
>>> APIs. The toolstack wouldn't be able to do anything about passing
>>> through a given device without knowing which device it should be
>>> passing
>>> through.
>>>
>>> Perhaps this info needs plumbing through to some new bit of the
>>> toolstack, but it is surely available somewhere.
>>>
>>> If you meant the virtual SBDF then that is in libxl_device_pci.vdevfn.
>> I added prints in libxl__device_pci_add. vdevfn is always 0 so this may
>> not be the right variable to use.
>> Can you please recheck.
>>
>> Also the vdev-X entry in xenstore appears to be created from pciback
>> code and not from xl.
>> Check function xen_pcibk_publish_pci_dev.
>>
>> So I have to send a hypercall from pciback only.
> I don't think the necessarily follows.
>
> You could have the tools read the vdev-X node back on plug.
I have been trying to get the flow of caller of libxl__device_pci_add 
during pci device assignemnt from cfg file(cold boot).
It should be called form xl create flow. Is it called from C code or 
Python code.

libxl__device_pci_add calls xc_assign_device


Secondly, the vdev-X entry is created async by dom0 watching on event. 
So how the tools could read back and call assign device again.

static void xen_pcibk_be_watch(struct xenbus_watch *watch,
                              const char **vec, unsigned int len)
{
      ...
         switch (xenbus_read_driver_state(pdev->xdev->nodename)) {
         case XenbusStateInitWait:
                 xen_pcibk_setup_backend(pdev);
                 break;
}
>
> Or you could change things such that vdevfn is always chosen by the
> toolstack for ARM, not optionally like it is on x86.
>
> Please evaluate the options.
>
> Ian.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel