Re: [PATCH DOCDAY] xen: write a high level description of the sub-arch choices for heap layout

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 30/09/15 11:22, Ian Campbell wrote:
> The 3 options which (sub)arches have for the layout of their heaps is
> a little subtle (in particular the two CONFIG_SEPARATE_XENHEAP=n
> submodes) and can be a bit tricky to derive from the code.
>
> Therefore try and write down some guidance on what the various modes
> are.
>
> Note that this is intended more as a high level overview rather than a
> detailed guide to the full page allocator interfaces.
>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  xen/common/page_alloc.c | 97 +++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 97 insertions(+)
>
> diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
> index 2b8810c..46c1ab9 100644
> --- a/xen/common/page_alloc.c
> +++ b/xen/common/page_alloc.c
> @@ -20,6 +20,103 @@
>   * along with this program; If not, see <http://www.gnu.org/licenses/>.
>   */
>  
> +/*
> + * In general Xen maintains two pools of memory:
> + *
> + * - Xen heap: Memory which is always mapped (i.e accessible by
> + *             virtual address), via a permanent and contiguous
> + *             "direct mapping". Macros like va() and pa() are valid
> + *             for such memory.

Possibly worth stating that it safe to stash pointers into xenheap memory.

> + *
> + *             Xen heap pages are always anonymous (that is, not tied
> + *             or accounted to any particular domain).
> + *
> + * - Dom heap: Memory which must be explicitly mapped, usually
> + *             transiently with map_domain_page, in order to be
> + *             used. va() and pa() are not valid for such memory.

While stashing pointers into domheap memory is definitely buggy.

> + *
> + *             Dom heap pages are often tied to a particular domain,
> + *             but need not be (passing domain==NULL results in an
> + *             anonymous dom heap allocation).
> + *
> + * The exact nature of this split is a (sub)arch decision which can
> + * select one of three main variants:
> + *
> + * CONFIG_SEPARATE_XENHEAP=y
> + *
> + *   The xenheap is maintained as an entirely separate heap.
> + *
> + *   Arch code arranges for some (perhaps small) amount of physical
> + *   memory to be covered by a direct mapping and registers that
> + *   memory as the Xen heap (via init_xenheap_pages()) and the
> + *   remainder as the dom heap.
> + *
> + *   This mode of operation is most commonly used by 32-bit arches
> + *   where the virtual address space is insufficient to map all RAM.
> + *
> + * CONFIG_SEPARATE_XENHEAP=n W/ DIRECT MAP OF ALL RAM
> + *
> + *   All of RAM is covered by a permanent contiguous mapping and there
> + *   is only a single heap.
> + *
> + *   Memory allocated from the Xen heap is flagged (in
> + *   page_info.count_info) with PGC_xen_heap which may be used to
> + *   check and enforce correct usage of va()/pa() etc. Memory
> + *   allocated from the Dom heap must still be explicitly mapped
> + *   before use (e.g. with map_domain_page) in particular in common
> + *   code.
> + *
> + *   xenheap_max_mfn() should not be called by arch code.
> + *
> + *   This mode of operation is most commonly used by 64-bit arches
> + *   which have sufficient free virtual address space to permanently
> + *   map the largest practical amount RAM currently expected on that
> + *   arch.
> + *
> + * CONFIG_SEPARATE_XENHEAP=n W/ ONLY DIRECT MAP OF ONLY PARTIAL RAM
> + *
> + *   There is a single heap, but only the beginning (up to some
> + *   threshold) is covered by a permanent contiguous mapping.

Perhaps avoid the use of "beginning" here?  It is just an implementation
detail of the only current example.

In some copious free time, I want to see about striding the x86
directmap across NUMA nodes (to allow NUMA-local xenheap allocations
even on large boxes), at which point it won't be linear from the start.

~Andrew