Re: [RFC][PATCH 4/5] tools:firmware:hvmloader: reserve RMRR mappings in e820

["Chen, Tiejun" <tiejun.chen@intel.com>]

On 2014/8/14 3:10, Tian, Kevin wrote:
>> From: Chen, Tiejun
>> Sent: Tuesday, August 12, 2014 5:57 PM
>>
>> On 2014/8/12 20:25, Jan Beulich wrote:
>>>>>> On 12.08.14 at 12:59, <tiejun.chen@intel.com> wrote:
>>>> On 2014/8/12 0:00, Tian, Kevin wrote:
>>>>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>>>>> Sent: Sunday, August 10, 2014 11:53 PM
>>>>>>>>> On 08.08.14 at 23:47, <kevin.tian@intel.com> wrote:
>>>>>>> strictly speaking besides reserving in e820, you should also poke later
>>>>>>> MMIO BAR allocations to avoid confliction too. Currently it's relative
>>>>>>> to low_mem_pgend, which is likely to be different from host layout
>>>>>>> so it's still possible to see a virtual MMIO bar base conflicting to the
>>>>>>> RMRR ranges which are supposed to be sparse.
>>>>>>
>>>>>> Correct. And what's worse: Possible collisions between RMRRs and
>>>>>> the BIOS we place into the VM need to be taken care of, which may
>>>>>> turn out rather tricky.
>>>>>>
>>>>>
>>>>> right that becomes tricky. We can provide another hypercall to allow a
>>>>> VM tell Xen which RMRR can't be assigned due to confliction with gust
>>>>> BIOS or other hvmloader allocation (if confliction can't be resolved).
>>>>>
>>>>> If Xen detects a device owning RMRR is already assigned to the VM,
>>>>> then fail the hypercall and hvmloader just panic with information to
>>>>> indicate confliction.
>>>>>
>>>>> Otherwise Xen records the information and future dynamic device
>>>>> assignment like hotplug will be failed if associated RMRR will be in
>>>>> the confliction list.
>>>>
>>>>    From my point of view its becoming over complicated.
>>>>
>>>> In HVM case, theoretically any devices involving RMRR may be assigned to
>>>> any given VM. So it may not be necessary to introduce such complex
>>>> mechanism. Therefore, I think we can reserve all RMRR maps simply in
>>>> e820, and check if MMIO is overlapping with RMRR for every VM. It should
>>>> be acceptable.
>>>
>>> Then you didn't understand what Kevin and I said above. Just
>>
>> I have to admit I'm poor in this coverage.
>>
>>> keep in mind that the RMRRs can conflict not just with MMIO
>>> ranges inside the guest, but also RAM ranges (which include, as
>>> mentioned above, the range where the BIOS for the guest gets
>>> put).
>>>
>>> Jan
>>>
>>
>> So just to clarify, as a summary there are four ranges we should be
>> addressed:
>>
>> #1 MMIO in guest
>>
>> In my patch [RFC][v2][PATCH 5/6] tools:libxc: check if mmio BAR is out
>> of RMRR mappings,
>>
>> I will check if this is overlapping.
>
> hvmloader controls actual mmio BAR allocation, so it's important to have

I guess you're saying pci_setup().

After setup_guest(), in pci_setup() we will reallocate mmio and ram if 
necessary and possible. Then all final info is reflected to fill into GS 
e820.

> check there. And your patch treats the whole mmio as one big region
> to check overlapping with RMRR which is too coarse-grained. Better to check

But its easy to feasible.

> overlapping every time when an allocation, either of memory ranges, or
> MMIO ranges, actually happen.

What is your policy to handle a conflict?

I mean those RMRR mapping entries are undermined and often they are not 
continuous. For example, IGD needs two entries in my current BDW,

#1 ab805000 ~ ab819000
#2 ad000000 ~ af800000

So if just one of them conflicts something, how to handle such a case? 
Push mmio out of RMRR? Or allow many mmio hole?  As you know IGD can't 
work as long as one of two entries is overlapping.

So I think it may not be necessary to handle this as complicated mechanism.

 From my point of view its enough to double check RMRR in GS e820 since 
just do check rather than check-to-fix. If any overlap occurs we will 
post WARNING/ERROR to notify the user, then let user decide what we 
should do next. If they know don't need any PCI passthrough its fine. 
And especially, actually RMRR should be rare.

>
>>
>> #2 RAM in guest
>>
>> tools/firmware/hvmloader/e820.c:
>>       e820[nr].addr = 0x100000;
>>       e820[nr].size = (hvm_info->low_mem_pgend << PAGE_SHIFT) -
>> e820[nr].addr;
>
> Note memory allocation actually happens in libxc, where when you see a
> populate_physmap occurs it actually means a real allocation in guest
> physical address space. That's what you want to detect and avoid overlapping
> in the 1st place. hvmloader builds e820 assuming the same policy as libxc
> where it's the 2nd place to check (it's not a good design between hvmloader
> and libxc, ideally the memory e820 ranges should be passed from libxc)
>

Same concern like the above.

Thanks
Tiejun

>>
>> #3 Guest BIOS itself
>>
>> tools/firmware/hvmloader/e820.c:
>>       e820[nr].addr = bios_image_base;
>>
>> For #2 and #3 in my patch [RFC][v2][PATCH 3/6] tools:firmware:hvmloader:
>> reserve RMRR mappings in e820, we will check if RMRR is overlapping
>> these ranges.
>>
>> #4 Machine RAM range for a given guest
>>
>> In this point I think RMRR already is as reserved in host e820, so its
>> not possible to allocate any RMRR as physical RAM to a VM.
>
> yes this is not the concern in this topic.
>
>>
>> If I'm still misunderstanding please correct me.
>>
>> Thanks
>> Tiejun
>