Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

["Chen, Tiejun" <tiejun.chen@intel.com>]

On 2014/11/4 15:54, Jan Beulich wrote:
>>>> On 04.11.14 at 06:05, <tiejun.chen@intel.com> wrote:
>> On 2014/11/3 20:34, Jan Beulich wrote:
>>>>>> On 03.11.14 at 12:58, <tiejun.chen@intel.com> wrote:
>>>> Firstly we have a rule that we just allow all devices associated one
>>>> RMRR to be assign same VM, right? So I mean while we create VM, we
>>>> always call current hypercall but inside hypercall, Xen can know which
>>>> devices will be assigned to this VM.
>>>
>>> I.e. the hypercall (at least optionally) becomes per-domain rather
>>> than global. And you imply that device assignment happens
>>> before memory getting populated (which likely can be arranged
>>
>> I tried to find a clue about this point but unfortunately I can't trace
>> when we assign device exactly. But in theory, based on your hint I
>> prefer the device assignment should follow memory getting populated.
>> Because when we add a device, we need to create iommu map so this means
>> at this moment the guest should already finish populating memory, right?
>
> There's no such strong connection: When a device gets assigned,
> IOMMU mappings get created for all memory the guest already has
> assigned (which at least in theory can include the "none" case).
> When (more) memory gets assigned after a device was already
> assigned to the guest, the IOMMU mappings would simply get
> updated.
>
> While I think you're right in that memory assignment happens
> before device assignment, for your specific purpose it might have

Yeah, I also double checked this sequence with printk.

> been easier the other way around, since when memory gets
> populated first you'll need special peeking into which devices will
> get assigned later in order to avoid the respective RMRR areas,
> or you'll need to modify device assignment code to move the
> RAM populated there out of the way.
>

Although I really don't grab these device assignment codes explicitly, 
but looks it may corrupt some existing frameworks to move device 
assignment codes.

I'm not sure if you already have a definitive solution. If yes, please 
guide me and just ignore the reset.

But if not, I have a preliminary picture as follows:

#1 Policy

We will introduce a parameter globally, 'pci_rdmforce'. Then we can pass 
that in the config file, like

pci_rdmforce = 1 => Of course this should be 0 by default.

'1' means we should force check to reserve all ranges. If failed VM 
wouldn't be created successfully. This also can give user a chance to 
work well with later hotplug, even if not a device assignment while 
creating VM.

But we can override that by one specific pci device:

pci = ['AA:BB.CC,rdmforce=0/1]

But this 'rdmforce' should be 1 by default since obviously any 
passthrough device always need to do this. Actually no one really want 
to set as '0' so it may be unnecessary but I'd like to leave this as a 
potential approach.

So if 'pci_rdmforce = 0' and 'rdmforce = 1', we just check to reserve 
the ranges associated to this device.

And in any case of 'pci_rdmforce = 0' which means we have nothing to do, 
so we will have a higher likelihood of failure to assign a device with 
RMRR in both case of adding and attaching it. So we have to reject that 
in case of overlapping. Actually this just depends on whether we can set 
these identified p2m mapping.

#2 How-to

Firstly we should post actively a recommendation message to indicate 
we'd better set 'pci_rdmforce' to work passthrough, if any RMRR exists 
while parsing ACPI.

Next, we need a new domctl to provide these info, 'pci_rdmforce' and 
'rdm_force' when parse the config file. Certainly we may need to 
introduce and set two new fields in strcut domain, and since then we 
just use these fields to modify our current existing iommu callback 
inside to support our policy. I mean we just expose those associated 
RMRR entry. I think its easy to implement this since inside Xen we can 
know which entry is owned by which device. So this can benefit us to 
avoid modifying any tools codes and most Xen codes we already addressed.

So is it feasible to you? Or Anything else I'm missing?

Thanks
Tiejun