Re: [PATCH v2 00/21] xen/arm: Add support for non-pci passthrough

[Julien Grall <julien.grall@linaro.org>]

Hi Ian,

On 10/09/14 03:51, Ian Campbell wrote:
> On Wed, 2014-09-10 at 11:22 +0200, Christoffer Dall wrote:
>> On Tue, Sep 9, 2014 at 9:34 PM, Julien Grall <julien.grall@linaro.org> wrote:
>>> (Adding Christoffer)
>>>
>>> Hi Ian,
>>>
>>> On 09/09/14 07:34, Ian Campbell wrote:
>>>>
>>>> On Thu, 2014-07-31 at 16:00 +0100, Julien Grall wrote:
>>>>>
>>>>>       - Only common device properties (interrupts, regs) are written to
>>>>>       the guest device tree. Device that needs other properties may not
>>>>> work.
>>>>
>>>>
>>>> So I've glanced through the later (more toolstack oriented) bits from
>>>> towards the end but I think there's a question of the target users which
>>>> needs thinking about before I can have a sensible opinion on those.
>>>>
>>>> As I see it the main purpose of this series is to get the underlying
>>>> plumbing in place (wiring up iommus, routing IRQs etc) to support guests
>>>> with passthrough devices, for embedded folks to use and to provide a
>>>> basis for eventual PCI passthrough functionality. I really want to see
>>>> this stuff in 4.5
>>>>
>>>> What I'm concerned about is the toolstack side. TBH I'm not very keen on
>>>> the thing with exposing very DT specific stuff like compatible strings
>>>> down from the hypervisor via domctls.
>>>> It's not really clear how best to expose this functionality, I have a
>>>> feeling that this series either goes too far or not far enough and ends
>>>> up not really satisfying anyone.
>>>
>>>
>>> I don't see many other solutions to get the compatible strings. There is no
>>> easy way to get the property from DOM0, unless we introduce a new driver in
>>> Linux.
>>>
>>
>> The toolstack you are using to create your guest must necessarily know
>> which guest it is creating, including device properties of a device a
>> user wishes to assign  It can know this because it's hardcoded,
>> included in some config files, or supplied directly by the user.  I
>> think you really want to decouple the hardware description method for
>> Dom0 from retrieving resource description about your device. Can't you
>> simply reference the device to Linux through its sysfs handle and use
>> your Xen-passthrough-layer-hypercall-magic (which I know nothing
>> about) to have Dom0 tell Xen to map/route the relevant resources?
>
> By Xen-passthrough-layer-hypercall-magic do you mean the thing which
> lets the userspace toolstack make hypercalls (which is called "privcmd"
> FWIW) or are you talking about some specific passtrhough related kernel
> driver (like VFIO? which has no Xen equivalent right now)
>
> If you mean the former then I think Julien's code already does this --
> it makes hypercalls telling Xen to map certain MMIO regions to guests.
> What's in question is where the inputs to those hypercalls came from.
>
>>>> My suspicion is that regular folks won't really be using passthrough
>>>> until it is via PCI and that in the meantime this functionality is only
>>>> going to be used by e.g. people building embedded system and superkeen
>>>> early adopters both of whom know what they are doing and can tolerate
>>>> some hacks etc to get things working (and I think that's fine, it's
>>>> still a worthwhile set of things to get into 4.5 and those folks are
>>>> worth supporting).
>>>>
>>>> I'm also worried that we may be committing ourselves to a libxl API
>>>> already without really working through all the issues (e.g. other
>>>> properties).
>>>>
>>>> Given that I wonder if we wouldn't be better off for 4.5 supporting
>>>> something much simpler at the toolstack level, namely allowing users to
>>>> use iomem= and irq= in their domain config to map platform devices
>>>> through (already works with your series today?)
>>>
>>>
>>> This would need a bit a plumbing for irq part to allow the user choosing the
>>> VIRQ (like Arianna did for MMIO range).
>>>
>>
>> My Xen knowledge is limited here.  Is the iomem= and irq= commands
>> given to your Dom0 toolstack, Dom0, or Xen itself?
>
> They are things that the user can write into the guest cfg file,
> containing lists of the relevant resources. e.g. to pass IRQ 42 to the
> guest: irqs = [ 42 ]
>
> The toolstack parses that and makes hypercalls while building a guest to
> tell Xen to map those regions through.
>
>>    How does a user
>> know which physical address in Xen's physical address space needs to
>> be remapped based on the hardware description language for Dom0?
>
> This is an interesting question. The short answer for platform device
> type things is "they just know" (they presumably have datasheets etc).
> But I think the underlying question here is whether they are given in PA
> space or dom0 IPA space, right?
>
> The way that this works on x86 is that dom0 sees the real underlying
> MMIO addresses in e.g. PCI BARs and /proc/iomem etc. So the hypercalls
> to map MMIO regions to guests all take real physical addresses and
> nothing has to worry about IPA vs PA issues.
>
> On ARM things are potentially a bit more complex because dom0 is running
> with second stage paging. However we always map the MMIO regions 1:1 for
> dom0, and I think we will always have to do that (the "1:1 workaround"
> refers to RAM regions only). So I think we can continue to treat these
> things as proper physical addresses and don't need to introduce variants
> of the hypercalls which work in terms of IPAs (or you could argue that
> they already do and the translation is currently a nop).

The MMIO regions for platform device to passthrough are not mapped into 
DOM0 memory. Although, the device is still described in the device tree.

I'm not sure if it's useful for map the MMIO in DOM0 for a simple 
passthrough as for now we don't have any "platform-back" drivers in DOM0 
that will reset the device.

Maybe we should stay consistent with the IRQ assignment, as we can't 
assign to DOM0 for now (this is because there is not driver to unmap them).

Regards,

-- 
Julien Grall