Re: Xenheap placement in a NUMA system

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 28/09/15 13:28, Mario Lodde wrote:
> Hi everyone,
>
> I’m new to Xen and I’m trying to understand how physical memory is managed. If I’m understanding correctly the source code, the hypervisor allocates the memory it needs only from the xenheap, while the rest of the memory is used by dom0 and the domUs.

To a first approximation, yes.

The difference between the xenheap and domheap is that the xenheap is
always mapped into virtual address space, while the domheap might not be.

It is therefore safe to create a pointer to an item in xenheap memory,
but domheap pages may only be accessed using {,un}map_domheap_page()
(and friends).

> If we consider a NUMA system, is the entire xenheap concentrated in the boot node or is it spread across the nodes?

This is architecture dependent.  On x86, the xenheap limit is at the
directmap end, which puts it at the 5TiB physical boundary.

On a small machine (< 5TB of RAM), xenheap pages will be available
across all NUMA nodes.  On larger machines, this will not be the case.

> In case it’s all stored in the boot node, and dom0’s memory is also contained entirely in the same node, is it correct to say that a failure in another node would just require to shut down the VMs that are running in that node but the rest of the system is still able to run?

What do you mean by "failure" here?  A full DIMM failure would almost
certainly be completely fatal to Xen, while an individual cell failure
does have logic for being propagated to the owning domain, in the hope
that it can take some corrective action.

~Andrew