Re: [v7][RFC][PATCH 06/13] hvmloader/ram: check if guest memory is out of reserved device memory maps

["Chen, Tiejun" <tiejun.chen@intel.com>]

On 2014/11/3 19:43, Jan Beulich wrote:
>>>> On 03.11.14 at 12:32, <tiejun.chen@intel.com> wrote:
>> On 2014/11/3 17:51, Jan Beulich wrote:
>>>>>> On 03.11.14 at 10:40, <tiejun.chen@intel.com> wrote:
>>>> On 2014/11/3 16:56, Jan Beulich wrote:
>>>>>>>> On 03.11.14 at 06:49, <tiejun.chen@intel.com> wrote:
>>>>>> On 2014/10/31 16:20, Jan Beulich wrote:
>>>>>>>>>> On 31.10.14 at 07:21, <kevin.tian@intel.com> wrote:
>>>>>>>>>      From: Chen, Tiejun
>>>>>>>>> Sent: Friday, October 31, 2014 1:41 PM
>>>>>>>>> On 2014/10/30 17:20, Jan Beulich wrote:
>>>>>>>>>> Thinking about this some more, this odd universal hole punching in
>>>>>>>>>> the E820 is very likely to end up causing problems. Hence I think
>>>>>>>>>> this really should be optional behavior, with pass through of devices
>>>>>>>>>> associated with RMRRs failing if not used. (This ought to include
>>>>>>>>>> punching holes for _just_ the devices passed through to a guest
>>>>>>>>>> upon creation when the option is not enabled.)
>>>>>>>>>
>>>>>>>>> Yeah, we had a similar discussion internal to add a parameter to force
>>>>>>>>> reserving RMRR. In this case we can't create a VM if these ranges
>>>>>>>>> conflict with anything. So what about this idea?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Adding a new parameter (e.g. 'check-passthrough') looks the right
>>>>>>>> approach. When the parameter is on, RMRR check/hole punch is
>>>>>>>> activated at VM creation. Otherwise we just keep existing behavior.
>>>>>>>>
>>>>>>>> If user configures device pass-through at creation time, this parameter
>>>>>>>> will be set by default. If user wants the VM capable of device hot-plug,
>>>>>>>> an explicit parameter can be added in the config file to enforce RMRR
>>>>>>>> check at creation time.
>>>>>>>
>>>>>>> Not exactly, I specifically described it slightly differently above. When
>>>>>>> devices get passed through and the option is absent, holes should be
>>>>>>> punched only for the RMRRs associated with those devices (i.e.
>>>>>>> ideally none). Of course this means we'll need a way to associate
>>>>>>> RMRRs with devices in the tool stack and hvmloader, i.e. the current
>>>>>>> XENMEM_reserved_device_memory_map alone won't suffice.
>>>>>>
>>>>>> Yeah, current hypercall just provide RMRR entries without that
>>>>>> associated BDF. And especially, in some cases one range may be shared by
>>>>>> multiple devices...
>>>>>
>>>>> Before we decide who's going to do an eventual change we need to
>>>>> determine what behavior we want, and whether this hypercall is
>>>>> really the right one. Quite possibly we'd need a per-domain view
>>>>> along with the global view, and hence rather than modifying this one
>>>>> we may need to introduce e.g. a new domctl.
>>>>>
>>>>
>>>> If we really need to work with a hypercall, maybe we can introduce a
>>>> little bit to construct that to callback with multiple entries like
>>>> this, for instance,
>>>>
>>>> RMRR entry0 have three devices, and entry1 have two devices,
>>>>
>>>> [start0, nr_pages0, bdf0],
>>>> [start0, nr_pages0, bdf1],
>>>> [start0, nr_pages0, bdf2],
>>>> [start1, nr_pages1, bdf3],
>>>> [start1, nr_pages1, bdf4],
>>>>
>>>> Although its cost more buffers, actually as you know this actual case is
>>>> really rare. So maybe this way can be feasible. Then we don't need
>>>> additional hypercall or xenstore.
>>>
>>> Conceptually, as a MEMOP, it has no business reporting BDFs. And
>>> then rather than returning the same address range more than once,
>>> having the caller supply a handle to an array and storing all of the
>>> SBDFs (or perhaps a single segment would suffice along with all the
>>> BDFs) there would seem to be an approach more consistent with
>>> what we do elsewhere.
>>
>> Here I'm wondering if we really need to expose BDFs to tools. Actually
>> tools just want to know those range no matter who owns these entries. I
>> mean we can do this in Xen.
>
> As pointed out before, in order for the tools to (a) avoid punching
> _all possible_ holes when not asked to but (b) punch holes for all
> devices assigned right at guest boot time, I don't think the tools
> can get away without having some way of identifying the _specific_
> reserved memory regions a guest needs. Whether this works via
> SBDF or some other means is tbd.
>
>> When we try to assign device as passthrough, Xen can get that bdf so Xen
>> can pre-check everything inside that hypercall, and Xen can return
>> something like this,
>>
>> #1 If this device has RMRR, we return that rmrr buffer. This is similar
>> with our current implementation.
>
> "That rmrr buffer" being which? Remember that there may be
> multiple devices associated with RMRRs and multiple devices
> assigned to a guest.

Maybe I don't describe what I want to do.

I know there are multiple device associated one RMRR, and multiple 
device can be assigned one guest.

Firstly we have a rule that we just allow all devices associated one 
RMRR to be assign same VM, right? So I mean while we create VM, we 
always call current hypercall but inside hypercall, Xen can know which 
devices will be assigned to this VM. So Xen still lookup that RMRR list 
but now Xen would check if these RMRR belongs to that device we want to 
assign this domain. If yes, we just let that callback go through these 
RMRR info from that list but exclude other unrelated RMRR. If not, we 
don't go through any RMRR info so that 'nr_entries' is also zero.

Thanks
Tiejun

>
>> #2 If not, we return 'nr_entries' as '0' to notify hvmloader it has
>> nothing to do.
>
> This (as vague as it is) hints in the same domain-specific reserved
> region determination that I was already hinting at.
>
> Jan
>
>