Re: [PATCH V3 1/1] expand x86 arch_shared_info to support >3 level p2m tree

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 12/09/2014 11:31, Juergen Gross wrote:
> On 09/09/2014 12:49 PM, Juergen Gross wrote:
>> On 09/09/2014 12:27 PM, Andrew Cooper wrote:
>>> On 09/09/14 10:58, Juergen Gross wrote:
>>>> The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list
>>>> currently contains the mfn of the top level page frame of the 3 level
>>>> p2m tree, which is used by the Xen tools during saving and restoring
>>>> (and live migration) of pv domains. With three levels of the p2m tree
>>>> it is possible to support up to 512 GB of RAM for a 64 bit pv domain.
>>>> A 32 bit pv domain can support more, as each memory page can hold 1024
>>>> instead of 512 entries, leading to a limit of 4 TB. To be able to
>>>> support more RAM on x86-64 an additional level is to be added.
>>>>
>>>> This patch expands struct arch_shared_info with a new p2m tree root
>>>> and the number of levels of the p2m tree. The new information is
>>>> indicated by the domain to be valid by storing ~0UL into
>>>> pfn_to_mfn_frame_list_list (this should be done only if more than
>>>> three levels are needed, of course).
>>>
>>> A small domain feeling a little tight on space could easily opt for a 2
>>> or even 1 level p2m.  (After all, one advantage of virt is to cram many
>>> small VMs into a server).
>>>
>>> How is xen and toolstack support for n-level p2ms going to be 
>>> advertised
>>> to guests?  Simply assuming the toolstack is capable of dealing with
>>> this new scheme wont work with a new pv guest running on an older Xen.
>>
>> Is it really worth doing such an optimization? This would save only very
>> few pages.
>>
>> If you think it should be done we can add another SIF_* flag to
>> start_info->flags. In this case a domain using this feature could not be
>> migrated to a server with old tools, however. So we would probably end
>> with the need to be able to suppress that flag on a per-domain base.
>
> Any further comments?
>
> Which way should I go?
>

There are two approaches, with different up/downsides

1) continue to use the old method, and use the new method only when 
absolutely required.  This will function, but on old toolstacks, suffer 
migration/suspend failures when the toolstack fails to find the p2m.

2) Provide a Xen feature flag indicating the presence of N-level p2m 
support.  Guests which can see this flag are free to use N-level, and 
guests which can't are not.

Ultimately, giving more than 512GB to a current 64bit PV domain is not 
going to work, and the choice above depends on which failure mode you 
wish a new/old mix to have.

~Andrew