[PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Ian Campbell]

The 3 options which (sub)arches have for the layout of their heaps is
a little subtle (in particular the two CONFIG_SEPARATE_XENHEAP=n
submodes) and can be a bit tricky to derive from the code.

Therefore try and write down some guidance on what the various modes
are.

Note that this is intended more as a high level overview rather than a
detailed guide to the full page allocator interfaces.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 xen/common/page_alloc.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)

diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 2b8810c..46c1ab9 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -20,6 +20,103 @@
  * along with this program; If not, see <http://www.gnu.org/licenses/>.
  */
 
+/*
+ * In general Xen maintains two pools of memory:
+ *
+ * - Xen heap: Memory which is always mapped (i.e accessible by
+ *             virtual address), via a permanent and contiguous
+ *             "direct mapping". Macros like va() and pa() are valid
+ *             for such memory.
+ *
+ *             Xen heap pages are always anonymous (that is, not tied
+ *             or accounted to any particular domain).
+ *
+ * - Dom heap: Memory which must be explicitly mapped, usually
+ *             transiently with map_domain_page, in order to be
+ *             used. va() and pa() are not valid for such memory.
+ *
+ *             Dom heap pages are often tied to a particular domain,
+ *             but need not be (passing domain==NULL results in an
+ *             anonymous dom heap allocation).
+ *
+ * The exact nature of this split is a (sub)arch decision which can
+ * select one of three main variants:
+ *
+ * CONFIG_SEPARATE_XENHEAP=y
+ *
+ *   The xenheap is maintained as an entirely separate heap.
+ *
+ *   Arch code arranges for some (perhaps small) amount of physical
+ *   memory to be covered by a direct mapping and registers that
+ *   memory as the Xen heap (via init_xenheap_pages()) and the
+ *   remainder as the dom heap.
+ *
+ *   This mode of operation is most commonly used by 32-bit arches
+ *   where the virtual address space is insufficient to map all RAM.
+ *
+ * CONFIG_SEPARATE_XENHEAP=n W/ DIRECT MAP OF ALL RAM
+ *
+ *   All of RAM is covered by a permanent contiguous mapping and there
+ *   is only a single heap.
+ *
+ *   Memory allocated from the Xen heap is flagged (in
+ *   page_info.count_info) with PGC_xen_heap which may be used to
+ *   check and enforce correct usage of va()/pa() etc. Memory
+ *   allocated from the Dom heap must still be explicitly mapped
+ *   before use (e.g. with map_domain_page) in particular in common
+ *   code.
+ *
+ *   xenheap_max_mfn() should not be called by arch code.
+ *
+ *   This mode of operation is most commonly used by 64-bit arches
+ *   which have sufficient free virtual address space to permanently
+ *   map the largest practical amount RAM currently expected on that
+ *   arch.
+ *
+ * CONFIG_SEPARATE_XENHEAP=n W/ ONLY DIRECT MAP OF ONLY PARTIAL RAM
+ *
+ *   There is a single heap, but only the beginning (up to some
+ *   threshold) is covered by a permanent contiguous mapping.
+ *
+ *   Memory allocated from the Xen heap is allocated from below the
+ *   threshold and flagged with PGC_xen_heap. Memory allocated from
+ *   the dom heap is allocated from anywhere in the heap (although it
+ *   will prefer to allocate from as high as possible to try and keep
+ *   Xen heap suitable memory available).
+ *
+ *   Arch code must call xenheap_max_mfn() to signal the limit of the
+ *   direct mapping.
+ *
+ *   This mode of operation is most commonly used by 64-bit arches
+ *   which have a restricted amount of virtual address space available
+ *   for a direct map (due to e.g. reservations for other purposes)
+ *   such that it is not possible to map all of RAM on systems with
+ *   the largest practical amount of RAM currently expected on that
+ *   arch.
+ *
+ * Boot Allocator
+ *
+ *   In addition to the two primary pools (xen heap and dom heap) a
+ *   third "boot allocator" is used at start of day. This is a
+ *   simplified allocator which can be used.
+ *
+ *   Typically all memory which is destined to be dom heap memory
+ *   (which is everything in the CONFIG_SEPARATE_XENHEAP=n
+ *   configurations) is first allocated to the boot allocator (with
+ *   init_boot_pages()) and is then handed over to the main dom heap in
+ *   end_boot_allocator().
+ *
+ * "Contiguous" mappings
+ *
+ *   Note that although the above talks about "contiguous" mappings
+ *   some architectures implement a scheme ("PDX compression") to
+ *   compress unused portions of the machine address space (i.e. large
+ *   gaps between distinct banks of memory) in order to avoid creating
+ *   enormous frame tables and direct maps which mostly map
+ *   nothing. Thus a contiguous mapping may still have distinct
+ *   regions within it.
+ */
+
 #include <xen/config.h>
 #include <xen/init.h>
 #include <xen/types.h>
-- 
2.1.4

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Jan Beulich]

>>> On 30.09.15 at 12:22, <ian.campbell@citrix.com> wrote:
> The 3 options which (sub)arches have for the layout of their heaps is
> a little subtle (in particular the two CONFIG_SEPARATE_XENHEAP=n
> submodes) and can be a bit tricky to derive from the code.
> 
> Therefore try and write down some guidance on what the various modes
> are.
> 
> Note that this is intended more as a high level overview rather than a
> detailed guide to the full page allocator interfaces.

Thanks for writing this up, just two minor things:

> + * CONFIG_SEPARATE_XENHEAP=n W/ DIRECT MAP OF ALL RAM
> + *
> + *   All of RAM is covered by a permanent contiguous mapping and there
> + *   is only a single heap.
> + *
> + *   Memory allocated from the Xen heap is flagged (in
> + *   page_info.count_info) with PGC_xen_heap which may be used to
> + *   check and enforce correct usage of va()/pa() etc. Memory

What is this "check and enforce" about? There are validation uses
of the flag, but I don't recall any in virt<->phys address translation.

> + *   allocated from the Dom heap must still be explicitly mapped
> + *   before use (e.g. with map_domain_page) in particular in common
> + *   code.
> + *
> + *   xenheap_max_mfn() should not be called by arch code.
> + *
> + *   This mode of operation is most commonly used by 64-bit arches
> + *   which have sufficient free virtual address space to permanently
> + *   map the largest practical amount RAM currently expected on that
> + *   arch.
> + *
> + * CONFIG_SEPARATE_XENHEAP=n W/ ONLY DIRECT MAP OF ONLY PARTIAL RAM

I think one of the two ONLY should be dropped.

Jan

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Ian Campbell]

On Wed, 2015-09-30 at 04:33 -0600, Jan Beulich wrote:
> > > > On 30.09.15 at 12:22, <ian.campbell@citrix.com> wrote:
> > The 3 options which (sub)arches have for the layout of their heaps is
> > a little subtle (in particular the two CONFIG_SEPARATE_XENHEAP=n
> > submodes) and can be a bit tricky to derive from the code.
> > 
> > Therefore try and write down some guidance on what the various modes
> > are.
> > 
> > Note that this is intended more as a high level overview rather than a
> > detailed guide to the full page allocator interfaces.
> 
> Thanks for writing this up, just two minor things:
> 
> > + * CONFIG_SEPARATE_XENHEAP=n W/ DIRECT MAP OF ALL RAM
> > + *
> > + *   All of RAM is covered by a permanent contiguous mapping and there
> > + *   is only a single heap.
> > + *
> > + *   Memory allocated from the Xen heap is flagged (in
> > + *   page_info.count_info) with PGC_xen_heap which may be used to
> > + *   check and enforce correct usage of va()/pa() etc. Memory
> 
> What is this "check and enforce" about? There are validation uses
> of the flag, but I don't recall any in virt<->phys address translation.

I think I misremembered/assumed.

Arm has an is_xen_heap_page check for the separate xenheap case.

I suppose s/may be/could be/ would be true (sort of) but I guess I'll just
drop that bit.

> > + *   allocated from the Dom heap must still be explicitly mapped
> > + *   before use (e.g. with map_domain_page) in particular in common
> > + *   code.
> > + *
> > + *   xenheap_max_mfn() should not be called by arch code.
> > + *
> > + *   This mode of operation is most commonly used by 64-bit arches
> > + *   which have sufficient free virtual address space to permanently
> > + *   map the largest practical amount RAM currently expected on that
> > + *   arch.
> > + *
> > + * CONFIG_SEPARATE_XENHEAP=n W/ ONLY DIRECT MAP OF ONLY PARTIAL RAM
> 
> I think one of the two ONLY should be dropped.

Agreed.

Ian.

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Andrew Cooper]

On 30/09/15 11:22, Ian Campbell wrote:
> The 3 options which (sub)arches have for the layout of their heaps is
> a little subtle (in particular the two CONFIG_SEPARATE_XENHEAP=n
> submodes) and can be a bit tricky to derive from the code.
>
> Therefore try and write down some guidance on what the various modes
> are.
>
> Note that this is intended more as a high level overview rather than a
> detailed guide to the full page allocator interfaces.
>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/common/page_alloc.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 97 insertions(+)
>
> diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
> index 2b8810c..46c1ab9 100644
> --- a/xen/common/page_alloc.c
> +++ b/xen/common/page_alloc.c
> @@ -20,6 +20,103 @@
>   * along with this program; If not, see <http://www.gnu.org/licenses/>.
>   */
>  
> +/*
> + * In general Xen maintains two pools of memory:
> + *
> + * - Xen heap: Memory which is always mapped (i.e accessible by
> + *             virtual address), via a permanent and contiguous
> + *             "direct mapping". Macros like va() and pa() are valid
> + *             for such memory.

Possibly worth stating that it safe to stash pointers into xenheap memory.

> + *
> + *             Xen heap pages are always anonymous (that is, not tied
> + *             or accounted to any particular domain).
> + *
> + * - Dom heap: Memory which must be explicitly mapped, usually
> + *             transiently with map_domain_page, in order to be
> + *             used. va() and pa() are not valid for such memory.

While stashing pointers into domheap memory is definitely buggy.

> + *
> + *             Dom heap pages are often tied to a particular domain,
> + *             but need not be (passing domain==NULL results in an
> + *             anonymous dom heap allocation).
> + *
> + * The exact nature of this split is a (sub)arch decision which can
> + * select one of three main variants:
> + *
> + * CONFIG_SEPARATE_XENHEAP=y
> + *
> + *   The xenheap is maintained as an entirely separate heap.
> + *
> + *   Arch code arranges for some (perhaps small) amount of physical
> + *   memory to be covered by a direct mapping and registers that
> + *   memory as the Xen heap (via init_xenheap_pages()) and the
> + *   remainder as the dom heap.
> + *
> + *   This mode of operation is most commonly used by 32-bit arches
> + *   where the virtual address space is insufficient to map all RAM.
> + *
> + * CONFIG_SEPARATE_XENHEAP=n W/ DIRECT MAP OF ALL RAM
> + *
> + *   All of RAM is covered by a permanent contiguous mapping and there
> + *   is only a single heap.
> + *
> + *   Memory allocated from the Xen heap is flagged (in
> + *   page_info.count_info) with PGC_xen_heap which may be used to
> + *   check and enforce correct usage of va()/pa() etc. Memory
> + *   allocated from the Dom heap must still be explicitly mapped
> + *   before use (e.g. with map_domain_page) in particular in common
> + *   code.
> + *
> + *   xenheap_max_mfn() should not be called by arch code.
> + *
> + *   This mode of operation is most commonly used by 64-bit arches
> + *   which have sufficient free virtual address space to permanently
> + *   map the largest practical amount RAM currently expected on that
> + *   arch.
> + *
> + * CONFIG_SEPARATE_XENHEAP=n W/ ONLY DIRECT MAP OF ONLY PARTIAL RAM
> + *
> + *   There is a single heap, but only the beginning (up to some
> + *   threshold) is covered by a permanent contiguous mapping.

Perhaps avoid the use of "beginning" here?  It is just an implementation
detail of the only current example.

In some copious free time, I want to see about striding the x86
directmap across NUMA nodes (to allow NUMA-local xenheap allocations
even on large boxes), at which point it won't be linear from the start.

~Andrew

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Ian Campbell]

On Wed, 2015-09-30 at 12:10 +0100, Andrew Cooper wrote:
> + *
> > + * CONFIG_SEPARATE_XENHEAP=n W/ ONLY DIRECT MAP OF ONLY PARTIAL RAM
> > + *
> > + *   There is a single heap, but only the beginning (up to some
> > + *   threshold) is covered by a permanent contiguous mapping.
> 
> Perhaps avoid the use of "beginning" here?  It is just an implementation
> detail of the only current example.

It's an implementation detail which is currently exposed to the arch code
via the need to use xenheap_max_mfn() (or not) and the shape of that API
though.

> In some copious free time, I want to see about striding the x86
> directmap across NUMA nodes (to allow NUMA-local xenheap allocations
> even on large boxes), at which point it won't be linear from the start.

In which case this bit of doc would need some adjustments over and above
avoiding the work beginning I think, at least to adjust to the replacement
for xenheap_max_mfn().

Ian.

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Andrew Cooper]

On 30/09/15 12:28, Ian Campbell wrote:
> On Wed, 2015-09-30 at 12:10 +0100, Andrew Cooper wrote:
>> + *
>>> + * CONFIG_SEPARATE_XENHEAP=n W/ ONLY DIRECT MAP OF ONLY PARTIAL RAM
>>> + *
>>> + *   There is a single heap, but only the beginning (up to some
>>> + *   threshold) is covered by a permanent contiguous mapping.
>> Perhaps avoid the use of "beginning" here?  It is just an implementation
>> detail of the only current example.
> It's an implementation detail which is currently exposed to the arch code
> via the need to use xenheap_max_mfn() (or not) and the shape of that API
> though.
>
>> In some copious free time, I want to see about striding the x86
>> directmap across NUMA nodes (to allow NUMA-local xenheap allocations
>> even on large boxes), at which point it won't be linear from the start.
> In which case this bit of doc would need some adjustments over and above
> avoiding the work beginning I think, at least to adjust to the replacement
> for xenheap_max_mfn().

True in both cases, in which case the original wording in fine.

~Andrew

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Ian Campbell]

On Wed, 2015-09-30 at 12:10 +0100, Andrew Cooper wrote:

> > + *
> > + *             Xen heap pages are always anonymous (that is, not tied
> > + *             or accounted to any particular domain).
> > + *
> > + * - Dom heap: Memory which must be explicitly mapped, usually
> > + *             transiently with map_domain_page, in order to be
> > + *             used. va() and pa() are not valid for such memory.
> 
> While stashing pointers into domheap memory is definitely buggy.

Is this true even considering the result of e.g. map_domain_page_global?

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Andrew Cooper]

On 30/09/15 12:31, Ian Campbell wrote:
> On Wed, 2015-09-30 at 12:10 +0100, Andrew Cooper wrote:
>
>>> + *
>>> + *             Xen heap pages are always anonymous (that is, not tied
>>> + *             or accounted to any particular domain).
>>> + *
>>> + * - Dom heap: Memory which must be explicitly mapped, usually
>>> + *             transiently with map_domain_page, in order to be
>>> + *             used. va() and pa() are not valid for such memory.
>> While stashing pointers into domheap memory is definitely buggy.
> Is this true even considering the result of e.g. map_domain_page_global?
>

Ah yes - constructing a pointer into something mapped as global is safe.

Basically I was wondering about some wording to state that things like:

p = map_domain_page();
d->foo->bar = p->baz;
unmap_domain_page(p);

is unsafe and shouldn't be done.  There is surprisingly little
difference between a xenheap page and a map_domain_page_global()'d page,
as they are both present in the permanent mappings.

~Andrew

Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Jan Beulich]

>>> On 30.09.15 at 13:31, <ian.campbell@citrix.com> wrote:
> On Wed, 2015-09-30 at 12:10 +0100, Andrew Cooper wrote:
> 
>> > + *
>> > + *             Xen heap pages are always anonymous (that is, not tied
>> > + *             or accounted to any particular domain).
>> > + *
>> > + * - Dom heap: Memory which must be explicitly mapped, usually
>> > + *             transiently with map_domain_page, in order to be
>> > + *             used. va() and pa() are not valid for such memory.
>> 
>> While stashing pointers into domheap memory is definitely buggy.
> 
> Is this true even considering the result of e.g. map_domain_page_global?

No. So if you wanted to adjust the wording, you'd want to exclude
that as well as vmap().

Jan