Re: [PATCH v2 16/17] libxc/xc_dom_arm: Copy ACPI tables to guest space

[Julien Grall <julien.grall@arm.com>]

On 21/07/2016 19:54, Stefano Stabellini wrote:
> On Thu, 21 Jul 2016, Julien Grall wrote:
>> Hi Stefano,
>>
>> On 21/07/16 18:53, Stefano Stabellini wrote:
>>> On Wed, 20 Jul 2016, Boris Ostrovsky wrote:
>>>> On 07/20/2016 01:28 PM, Stefano Stabellini wrote:
>>>>> On Wed, 20 Jul 2016, Boris Ostrovsky wrote:
>>>>>> On 07/20/2016 09:41 AM, Julien Grall wrote:
>>>>>>>
>>>>>>> On 20/07/2016 14:33, Boris Ostrovsky wrote:
>>>>>>>> On 07/20/2016 08:33 AM, Julien Grall wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 14/07/16 14:37, Stefano Stabellini wrote:
>>>>>>>>>> On Wed, 13 Jul 2016, Julien Grall wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> On 12/07/2016 17:58, Boris Ostrovsky wrote:
>>>>>>>>>>>> On 07/12/2016 12:10 PM, Julien Grall wrote:
>>>>>>>>>>>>> On 12/07/2016 16:08, Boris Ostrovsky wrote:
>>>>>>>>>>>>>> On 07/12/2016 10:57 AM, Shannon Zhao wrote:
>>>>>>>>>>>>> It will affect some others part of the guest if we don't
>>>>>>>>>>>>> increment
>>>>>>>>>>>>> the
>>>>>>>>>>>>> "maxmem" requested by the user. For ARM the ACPI blob
>>>>>>>>>>>>> will be
>>>>>>>>>>>>> exposed
>>>>>>>>>>>>> at a specific address that is outside of the guest RAM
>>>>>>>>>>>>> (see the
>>>>>>>>>>>>> guest
>>>>>>>>>>>>> memory layout in public/arch-arm.h).
>>>>>>>>>>>>>
>>>>>>>>>>>>> We chose this solution over putting in the RAM because
>>>>>>>>>>>>> the ACPI
>>>>>>>>>>>>> tables
>>>>>>>>>>>>> are not easily relocatable (compare to the device tree,
>>>>>>>>>>>>> initrd and
>>>>>>>>>>>>> kernel) so we could not take advantage of superpage in
>>>>>>>>>>>>> both stage-2
>>>>>>>>>>>>> (hypervisor) and stage-1 (kernel) page table.
>>>>>>>>>>>> Maybe this is something ARM-specific then. For x86 we will
>>>>>>>>>>>> want to
>>>>>>>>>>>> keep
>>>>>>>>>>>> maxmem unchanged.
>>>>>>>>>>> I don't think what I described in my previous mail is
>>>>>>>>>>> ARM-specific. The
>>>>>>>>>>> pressure will be more important on the TLBs, if Xen does not
>>>>>>>>>>> use
>>>>>>>>>>> superpage in
>>>>>>>>>>> the stage 2 page tables (i.e EPT for x86) no matter the
>>>>>>>>>>> architecture.
>>>>>>>>>>>
>>>>>>>>>>> IHMO, this seems to be a bigger drawback compare to add few
>>>>>>>>>>> more
>>>>>>>>>>> kilobytes to
>>>>>>>>>>> maxmem in the toolstack for the ACPI blob. You will loose
>>>>>>>>>>> them when
>>>>>>>>>>> creating
>>>>>>>>>>> the intermediate page table in any case.
>>>>>>>>>> I agree with Julien. On ARM we have to increase maxmem because
>>>>>>>>>> I don't
>>>>>>>>>> think that shattering a superpage is acceptable for just a few
>>>>>>>>>> KBs. In
>>>>>>>>>> fact, it's not much about increasing maxmem, but it's about
>>>>>>>>>> keeping
>>>>>>>>>> the
>>>>>>>>>> allocation of guest memory to the value passed by the user in
>>>>>>>>>> "memory",
>>>>>>>>>> so that it can be done in the most efficient way possible. (I
>>>>>>>>>> am
>>>>>>>>>> assuming users are going to allocate VMs of 2048MB, rather
>>>>>>>>>> than
>>>>>>>>>> 2049MB.)
>>>>>>>>>>
>>>>>>>>>> I wouldn't want to end up adding to the performance tuning
>>>>>>>>>> page on the
>>>>>>>>>> wiki "Make sure to add 1 more MB to the memory of your VM to
>>>>>>>>>> get the
>>>>>>>>>> most out of the system."
>>>>>>>>>>
>>>>>>>>>> I know that the location of the ACPI blob on x86 is different
>>>>>>>>>> in guest
>>>>>>>>>> memory space, but it seems to me that the problem would be the
>>>>>>>>>> same. Do
>>>>>>>>>> you have 1 gigabyte pages in stage-2 on x86? If so, I would
>>>>>>>>>> think
>>>>>>>>>> twice
>>>>>>>>>> about this. Otherwise, if you only have 4K and 2MB
>>>>>>>>>> allocations,
>>>>>>>>>> then it
>>>>>>>>>> might not make that much of a difference.
>>>>>>>>> Looking at the x86 code, 1 gigabyte pages seems to be supported.
>>>>>>>>>
>>>>>>>>> Boris, do you have any opinions on this?
>>>>>>>>
>>>>>>>> I don't think I understand the superpage shattering argument.  In
>>>>>>>> x86
>>>>>>>> the tables live in regular RAM and a guest is free to use those
>>>>>>>> addresses as regular memory.
>>>>>>>>
>>>>>>>> This apparently is different from how ARM manages the tables (you
>>>>>>>> said
>>>>>>>> in an earlier message that they are not part of RAM) so I can see
>>>>>>>> that
>>>>>>>> taking memory from RAM allocation to store the tables may affect
>>>>>>>> how
>>>>>>>> mapping is done, potentially causing GB pages to be broken.
>>>>>>>>
>>>>>>>> In fact (and I am totally speculating here) padding memory for x86
>>>>>>>> may
>>>>>>>> actually *cause* shattering because we will have (for example)
>>>>>>>> 2049MB of
>>>>>>>> RAM to deal with.
>>>>>>> Correct me if I am wrong. On your series you are populating the page
>>>>>>> at a specific address for the ACPI tables separately to the RAM
>>>>>>> allocation. So you will shatter GB pages if the user provides 2048MB
>>>>>>> because the ACPI tables is accounted in the 2048MB.
>>>>>> And to be honest I am not convinced this was a well selected address
>>>>>> (0xfc000000). I am actually thinking about moving it down (this may
>>>>>> require changing dsdt.asl). I don't know whether I will actually do it
>>>>>> in this version but it is certainly a possibility.
>>>>> I don't understand how this statement fits in the discussion.
>>>>>
>>>>> The memory allocation for the ACPI blob is done by the toolstack
>>>>> separately from the rest of guest memory, leading to two separate
>>>>> stage-2 pagetable allocations of less than 1-gigabyte pages. Is that
>>>>> correct?
>>>>
>>>>
>>>> If I move the table lower into memory we won't have to do any extra
>>>> allocation. The memory will have been already allocated for the guest,
>>>> we just map it and copy the tables.
>>>
>>> I see, thanks for the explanation. I think this could work for ARM too
>>> and should avoid the stage-2 shattering issue described above.
>>
>> But you will end up to have stage-1 shattering issue if you put the ACPI
>> tables lower into the guest RAM, reducing the overall performance. It is why I
>> first asked Shannon to put the ACPI outside of the guest RAM.
>
> I am not sure about this actually: even with the ACPI blob in the middle
> of guest memory, the guest OS could still use a single superpage for its
> own stage-1 memory mappings. I don't know if Linux does it that way, but
> it should be possible.

You are assuming that the guest will map the ACPI blob with the same 
attributes as the rest of the superpage.

IHMO, a sane operating system will want to map the ACPI blob read-only.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel