Re: Re: APIC rework

[Jeremy Fitzhardinge <jeremy@goop.org>]

On 11/25/09 06:10, Konrad Rzeszutek Wilk wrote:
> On Tue, Nov 24, 2009 at 03:35:22PM -0800, Jeremy Fitzhardinge wrote:
>   
>> On 11/24/09 11:44, Konrad Rzeszutek Wilk wrote:
>>     
>>>> At least dom0 parses this info from DSDT, so we can't have the assuption whether it is used or not, I think. And I also agree to add a new physdev_op to handle this case, and it should be better way to go.  
>>>> Based on this idea, I worked out the patch, attached!  In this patch, we introduced a new physdev_op PHYSDEVOP_setup_gsi for each GSI setup, and each domain can require to map each GSI in this case. 
>>>> In addition, I believe it is very safe to port the hypervisor patch to xen-3.4-x tree and keeps pv_ops dom0 running on it, since no logic is changed.  BTW, I also tested apic and non-apic cases, they works fine after applying the patches. 
>>>>     
>>>>         
>>> But I don't think you tested PCI front and PCI back.
>>>
>>> Mainly these lines worry me (can you inline the patch next time too, please):
>>>   
>>>       
>> (Inline+attach, or an inline attachment rather than plain inline, is
>> best.  Plain inline with quoted-printable encoding is awkward to deal with.)
>>
>>     
>>> +               map_irq.domid = DOMID_SELF;                                                                                                
>>> +               map_irq.type = MAP_PIRQ_TYPE_GSI;                                                                                          
>>> +               map_irq.index = gsi;                                                                                                       
>>> +               map_irq.pirq = irq;                                                                                                        
>>> +               rc = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);                                                                  
>>>
>>> For PCI passthrough to work, the domid needs to be for the guest domain, while
>>> in this case it is set to Dom0.
>>>
>>> There is already a method of extracting the domain id for PCI devices passed to
>>> the guest. Look in the 'xen_create_msi_irq' function.
>>>   
>>>       
>> Hm, I'm not very keen on having xen_create_msi_irq do its own traversal
>> of xenstore; it should take the domid as a parameter and its caller can
>> do the walk if necessary.  The direct call makes too much of a direct
>>     
> It would be the easiest way. Unfortunatly the call sequence is the
> following:
>
> pciback_do_op
>  pciback_enable_msi (dev)
>   ... arch_setup_msi_irqs
>   ...   arch_setup_msi_irq
>   ...     xen_setup_msi_irqs
>
> The caller of the pciback_enable_msi knows of the domain id, but the rest
> of the stack does not.  One way of doing this is to provide hooks in event.c, say:
>
>  [de|]register_msi_owner 
>
> (this is akin to how it was done in 2.6.18.hg)
>
> where the device is associated with a specific domain id. We would
> call those register/deregister when a new device has changed ownership
> to a domain.
>
> Back to the call trace. When we reach xen_setup_msi_irqs we can find the relation.
> Lookup the device (on this 'registration list') and see if it is assigned to a
> specific domain.
>
> But in essence it is similar in spirit to the mechanism of the xenbus_walk
> - that is to find out what domain id is associated with this device.
>
> With the patches posted by Xiantao this behavior would have to be
> emulated for PCI INTx devices as well, or some flavour of it.
>
> So the question is, Jeremy, would you prefer to have a '[de|]register_device_owner'
> call that both MSI and INTx can use and some way of traversing the list this
> creates and finding out if there is an owner for the device? Or just leave
> it with the xenbus_walk interface?
>   

I don't particularly object to xenbus_walk per-se, its just that I don't
think xen_create_msi_irq() should call it directly.  It would be OK for
xen_setup_msi_irqs() to call it and pass the results into
xen_create_msi_irq().

Would a registration list be any cleaner?  Presumably you'd just keep a
list of devices being controlled by other domains, so non-presence on
the list means DOMID_SELF, so you'd only need to worry about
registration on pciback paths.  If that could be done without needing to
touch common code (or do the horrible hacks that some of the earlier MSI
patches did), then maybe its worthwhile.

xenbus_walk() ends up actually talking to xenstored, so its fairly
expensive, and adds an opportunity for things to go wrong (it is
effectively doing a call out to dom0 usermode from the depths of the pci
code, which doesn't sound like a terribly healthy activity, but I guess
it has worked OK so far).

    J