Re: [RFC][PATCH 4/5] tools:firmware:hvmloader: reserve RMRR mappings in e820

["Chen, Tiejun" <tiejun.chen@intel.com>]

On 2014/8/15 7:11, Tian, Kevin wrote:
>> From: Chen, Tiejun
>> Sent: Wednesday, August 13, 2014 8:03 PM
>>
>> On 2014/8/14 3:10, Tian, Kevin wrote:
>>>> From: Chen, Tiejun
>>>> Sent: Tuesday, August 12, 2014 5:57 PM
>>>>
>>>> On 2014/8/12 20:25, Jan Beulich wrote:
>>>>>>>> On 12.08.14 at 12:59, <tiejun.chen@intel.com> wrote:
>>>>>> On 2014/8/12 0:00, Tian, Kevin wrote:
>>>>>>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>>>>>>> Sent: Sunday, August 10, 2014 11:53 PM
>>>>>>>>>>> On 08.08.14 at 23:47, <kevin.tian@intel.com> wrote:
>>>>>>>>> strictly speaking besides reserving in e820, you should also poke later
>>>>>>>>> MMIO BAR allocations to avoid confliction too. Currently it's relative
>>>>>>>>> to low_mem_pgend, which is likely to be different from host layout
>>>>>>>>> so it's still possible to see a virtual MMIO bar base conflicting to the
>>>>>>>>> RMRR ranges which are supposed to be sparse.
>>>>>>>>
>>>>>>>> Correct. And what's worse: Possible collisions between RMRRs and
>>>>>>>> the BIOS we place into the VM need to be taken care of, which may
>>>>>>>> turn out rather tricky.
>>>>>>>>
>>>>>>>
>>>>>>> right that becomes tricky. We can provide another hypercall to allow a
>>>>>>> VM tell Xen which RMRR can't be assigned due to confliction with gust
>>>>>>> BIOS or other hvmloader allocation (if confliction can't be resolved).
>>>>>>>
>>>>>>> If Xen detects a device owning RMRR is already assigned to the VM,
>>>>>>> then fail the hypercall and hvmloader just panic with information to
>>>>>>> indicate confliction.
>>>>>>>
>>>>>>> Otherwise Xen records the information and future dynamic device
>>>>>>> assignment like hotplug will be failed if associated RMRR will be in
>>>>>>> the confliction list.
>>>>>>
>>>>>>     From my point of view its becoming over complicated.
>>>>>>
>>>>>> In HVM case, theoretically any devices involving RMRR may be assigned
>> to
>>>>>> any given VM. So it may not be necessary to introduce such complex
>>>>>> mechanism. Therefore, I think we can reserve all RMRR maps simply in
>>>>>> e820, and check if MMIO is overlapping with RMRR for every VM. It
>> should
>>>>>> be acceptable.
>>>>>
>>>>> Then you didn't understand what Kevin and I said above. Just
>>>>
>>>> I have to admit I'm poor in this coverage.
>>>>
>>>>> keep in mind that the RMRRs can conflict not just with MMIO
>>>>> ranges inside the guest, but also RAM ranges (which include, as
>>>>> mentioned above, the range where the BIOS for the guest gets
>>>>> put).
>>>>>
>>>>> Jan
>>>>>
>>>>
>>>> So just to clarify, as a summary there are four ranges we should be
>>>> addressed:
>>>>
>>>> #1 MMIO in guest
>>>>
>>>> In my patch [RFC][v2][PATCH 5/6] tools:libxc: check if mmio BAR is out
>>>> of RMRR mappings,
>>>>
>>>> I will check if this is overlapping.
>>>
>>> hvmloader controls actual mmio BAR allocation, so it's important to have
>>
>> I guess you're saying pci_setup().
>>
>> After setup_guest(), in pci_setup() we will reallocate mmio and ram if
>> necessary and possible. Then all final info is reflected to fill into GS
>> e820.
>>
>>> check there. And your patch treats the whole mmio as one big region
>>> to check overlapping with RMRR which is too coarse-grained. Better to
>> check
>>
>> But its easy to feasible.
>>
>>> overlapping every time when an allocation, either of memory ranges, or
>>> MMIO ranges, actually happen.
>>
>> What is your policy to handle a conflict?
>>
>> I mean those RMRR mapping entries are undermined and often they are not
>> continuous. For example, IGD needs two entries in my current BDW,
>>
>> #1 ab805000 ~ ab819000
>> #2 ad000000 ~ af800000
>>
>> So if just one of them conflicts something, how to handle such a case?
>> Push mmio out of RMRR? Or allow many mmio hole?  As you know IGD can't
>> work as long as one of two entries is overlapping.
>>
>> So I think it may not be necessary to handle this as complicated mechanism.
>>
>>   From my point of view its enough to double check RMRR in GS e820 since
>> just do check rather than check-to-fix. If any overlap occurs we will
>> post WARNING/ERROR to notify the user, then let user decide what we
>> should do next. If they know don't need any PCI passthrough its fine.
>> And especially, actually RMRR should be rare.
>>
>
> I'm OK to do check-only w/o check-and-fix, at least it's a step forward
> to fail-safe.

Thanks a lot.

BTW, looks Xen have many known places, even bug, we need to clean or 
improve, right? So why don't we have an explicit plan to push this step 
by step? I mean at least we can document these somewhere like this,

#1 Notice some known problems
#2 Any useful discussion
#2 Workaround if possible
#3 Next step or plan

Its convenient to track them. Once one guy meet this again, he can find 
enough information to know how to deal with this.

Thanks
Tiejun

>
> Thanks
> Kevin
>