[Patch V3] support guest virtual mapped p2m list

[Patch V3] support guest virtual mapped p2m list

[Juergen Gross]

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list
currently contains the mfn of the top level page frame of the 3 level
p2m tree, which is used by the Xen tools during saving and restoring
(and live migration) of pv domains and for crash dump analysis. With
three levels of the p2m tree it is possible to support up to 512 GB of
RAM for a 64 bit pv domain.

A 32 bit pv domain can support more, as each memory page can hold 1024
instead of 512 entries, leading to a limit of 4 TB.

To be able to support more RAM on x86-64 switch to a virtual mapped
p2m list.

Changes in V3:
- removed XENFEAT_virtual_p2m completely as the linear p2m list and the
  3 level p2m tree can be used in parallel unless the domain size exceeds
  the limit mentioned above
- updated comments to reflect the parallel use of both p2m schemes

Changes in V2:
- add new structure member p2m_generation in arch_shared_info
- rename structure member referencing the p2m address space to p2m_cr3
- add some comments
- removed patches 2-4 as overriding missing XENFEAT_virtual_p2m will be
  done via kernel parameter (patch 2 will be resent after Xen 4.5 is out)


Juergen Gross (1):
  expand x86 arch_shared_info to support linear p2m list

 xen/include/public/arch-x86/xen.h | 36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

-- 
2.1.2

[Patch V3] expand x86 arch_shared_info to support linear p2m list

[Juergen Gross]

The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list
currently contains the mfn of the top level page frame of the 3 level
p2m tree, which is used by the Xen tools during saving and restoring
(and live migration) of pv domains and for crash dump analysis. With
three levels of the p2m tree it is possible to support up to 512 GB of
RAM for a 64 bit pv domain.

A 32 bit pv domain can support more, as each memory page can hold 1024
instead of 512 entries, leading to a limit of 4 TB.

To be able to support more RAM on x86-64 switch to an additional
virtual mapped p2m list.

This patch expands struct arch_shared_info with a new p2m list virtual
address, the root of the page table root and a p2m generation count.
The new information is indicated by the domain to be valid by storing
a non-zero value into the page table root member.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
 xen/include/public/arch-x86/xen.h | 36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff --git a/xen/include/public/arch-x86/xen.h b/xen/include/public/arch-x86/xen.h
index f35804b..c5e880b 100644
--- a/xen/include/public/arch-x86/xen.h
+++ b/xen/include/public/arch-x86/xen.h
@@ -220,11 +220,41 @@ typedef struct vcpu_guest_context vcpu_guest_context_t;
 DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
 
 struct arch_shared_info {
-    unsigned long max_pfn;                  /* max pfn that appears in table */
-    /* Frame containing list of mfns containing list of mfns containing p2m. */
+    /*
+     * Number of valid entries in the p2m table(s) anchored at
+     * pfn_to_mfn_frame_list_list and/or p2m_vaddr.
+     */
+    unsigned long max_pfn;
+    /*
+     * Frame containing list of mfns containing list of mfns containing p2m.
+     * A value of 0 indicates it has not yet been set up, ~0 indicates it has
+     * been set to invalid e.g. due to the p2m being too large for the 3-level
+     * p2m tree. In this case the linear mapper p2m list anchored at p2m_vaddr
+     * is to be used.
+     */
     xen_pfn_t     pfn_to_mfn_frame_list_list;
     unsigned long nmi_reason;
-    uint64_t pad[32];
+    /*
+     * Following three fields are valid if p2m_cr3 contains a value different
+     * from 0.
+     * p2m_cr3 is the root of the address space where p2m_vaddr is valid.
+     * p2m_cr3 is in the same format as a cr3 value in the vcpu register state
+     * and holds the folded machine frame number (via xen_pfn_to_cr3) of a
+     * L3 or L4 page table.
+     * p2m_vaddr holds the virtual address of the linear p2m list. All entries
+     * in the range [0...max_pfn[ are accessible via this pointer.
+     * p2m_generation will be incremented by the guest before and after each
+     * change of the mappings of the p2m list. p2m_generation starts at 0 and
+     * a value with the least significant bit set indicates that a mapping
+     * update is in progress. This allows guest external software (e.g. in Dom0)
+     * to verify that read mappings are consistent and whether they have changed
+     * since the last check.
+     * Modifying a p2m element in the linear p2m list is allowed via an atomic
+     * write only.
+     */
+    unsigned long p2m_cr3;         /* cr3 value of the p2m address space */
+    unsigned long p2m_vaddr;       /* virtual address of the p2m list */
+    unsigned long p2m_generation;  /* generation count of p2m mapping */
 };
 typedef struct arch_shared_info arch_shared_info_t;
 
-- 
2.1.2

Re: [Patch V3] expand x86 arch_shared_info to support linear p2m list

[Andrew Cooper]

On 05/12/14 09:35, Juergen Gross wrote:
> The x86 struct arch_shared_info field pfn_to_mfn_frame_list_list
> currently contains the mfn of the top level page frame of the 3 level
> p2m tree, which is used by the Xen tools during saving and restoring
> (and live migration) of pv domains and for crash dump analysis. With
> three levels of the p2m tree it is possible to support up to 512 GB of
> RAM for a 64 bit pv domain.
>
> A 32 bit pv domain can support more, as each memory page can hold 1024
> instead of 512 entries, leading to a limit of 4 TB.
>
> To be able to support more RAM on x86-64 switch to an additional
> virtual mapped p2m list.
>
> This patch expands struct arch_shared_info with a new p2m list virtual
> address, the root of the page table root and a p2m generation count.
> The new information is indicated by the domain to be valid by storing
> a non-zero value into the page table root member.
>
> Signed-off-by: Juergen Gross <jgross@suse.com>

Awesome - More documentation!

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

> ---
>  xen/include/public/arch-x86/xen.h | 36 +++++++++++++++++++++++++++++++++---
>  1 file changed, 33 insertions(+), 3 deletions(-)
>
> diff --git a/xen/include/public/arch-x86/xen.h b/xen/include/public/arch-x86/xen.h
> index f35804b..c5e880b 100644
> --- a/xen/include/public/arch-x86/xen.h
> +++ b/xen/include/public/arch-x86/xen.h
> @@ -220,11 +220,41 @@ typedef struct vcpu_guest_context vcpu_guest_context_t;
>  DEFINE_XEN_GUEST_HANDLE(vcpu_guest_context_t);
>  
>  struct arch_shared_info {
> -    unsigned long max_pfn;                  /* max pfn that appears in table */
> -    /* Frame containing list of mfns containing list of mfns containing p2m. */
> +    /*
> +     * Number of valid entries in the p2m table(s) anchored at
> +     * pfn_to_mfn_frame_list_list and/or p2m_vaddr.
> +     */
> +    unsigned long max_pfn;
> +    /*
> +     * Frame containing list of mfns containing list of mfns containing p2m.
> +     * A value of 0 indicates it has not yet been set up, ~0 indicates it has
> +     * been set to invalid e.g. due to the p2m being too large for the 3-level
> +     * p2m tree. In this case the linear mapper p2m list anchored at p2m_vaddr
> +     * is to be used.
> +     */
>      xen_pfn_t     pfn_to_mfn_frame_list_list;
>      unsigned long nmi_reason;
> -    uint64_t pad[32];
> +    /*
> +     * Following three fields are valid if p2m_cr3 contains a value different
> +     * from 0.
> +     * p2m_cr3 is the root of the address space where p2m_vaddr is valid.
> +     * p2m_cr3 is in the same format as a cr3 value in the vcpu register state
> +     * and holds the folded machine frame number (via xen_pfn_to_cr3) of a
> +     * L3 or L4 page table.
> +     * p2m_vaddr holds the virtual address of the linear p2m list. All entries
> +     * in the range [0...max_pfn[ are accessible via this pointer.
> +     * p2m_generation will be incremented by the guest before and after each
> +     * change of the mappings of the p2m list. p2m_generation starts at 0 and
> +     * a value with the least significant bit set indicates that a mapping
> +     * update is in progress. This allows guest external software (e.g. in Dom0)
> +     * to verify that read mappings are consistent and whether they have changed
> +     * since the last check.
> +     * Modifying a p2m element in the linear p2m list is allowed via an atomic
> +     * write only.
> +     */
> +    unsigned long p2m_cr3;         /* cr3 value of the p2m address space */
> +    unsigned long p2m_vaddr;       /* virtual address of the p2m list */
> +    unsigned long p2m_generation;  /* generation count of p2m mapping */
>  };
>  typedef struct arch_shared_info arch_shared_info_t;
>