Xenheap placement in a NUMA system

Xenheap placement in a NUMA system

[Mario Lodde]

Hi everyone,

I’m new to Xen and I’m trying to understand how physical memory is managed. If I’m understanding correctly the source code, the hypervisor allocates the memory it needs only from the xenheap, while the rest of the memory is used by dom0 and the domUs. 
If we consider a NUMA system, is the entire xenheap concentrated in the boot node or is it spread across the nodes? 
In case it’s all stored in the boot node, and dom0’s memory is also contained entirely in the same node, is it correct to say that a failure in another node would just require to shut down the VMs that are running in that node but the rest of the system is still able to run?

Thanks in advance for your help.

Best regards,
Mario

Re: Xenheap placement in a NUMA system

[Andrew Cooper]

On 28/09/15 13:28, Mario Lodde wrote:
> Hi everyone,
>
> I’m new to Xen and I’m trying to understand how physical memory is managed. If I’m understanding correctly the source code, the hypervisor allocates the memory it needs only from the xenheap, while the rest of the memory is used by dom0 and the domUs.

To a first approximation, yes.

The difference between the xenheap and domheap is that the xenheap is
always mapped into virtual address space, while the domheap might not be.

It is therefore safe to create a pointer to an item in xenheap memory,
but domheap pages may only be accessed using {,un}map_domheap_page()
(and friends).

> If we consider a NUMA system, is the entire xenheap concentrated in the boot node or is it spread across the nodes?

This is architecture dependent.  On x86, the xenheap limit is at the
directmap end, which puts it at the 5TiB physical boundary.

On a small machine (< 5TB of RAM), xenheap pages will be available
across all NUMA nodes.  On larger machines, this will not be the case.

> In case it’s all stored in the boot node, and dom0’s memory is also contained entirely in the same node, is it correct to say that a failure in another node would just require to shut down the VMs that are running in that node but the rest of the system is still able to run?

What do you mean by "failure" here?  A full DIMM failure would almost
certainly be completely fatal to Xen, while an individual cell failure
does have logic for being propagated to the owning domain, in the hope
that it can take some corrective action.

~Andrew

Re: Xenheap placement in a NUMA system

[Mario Lodde]

On 28 Sep 2015, at 15:29, Andrew Cooper <andrew.cooper3@citrix.com> wrote:

>> In case it’s all stored in the boot node, and dom0’s memory is also contained entirely in the same node, is it correct to say that a failure in another node would just require to shut down the VMs that are running in that node but the rest of the system is still able to run?
> 
> What do you mean by "failure" here?  A full DIMM failure would almost
> certainly be completely fatal to Xen, while an individual cell failure
> does have logic for being propagated to the owning domain, in the hope
> that it can take some corrective action.
> 
> ~Andrew

Yes, I mean losing  a whole DIMM on a node. I was wondering if it’s possible, assuming the xenheap and dom0 are confined to the boot node, to implement some sort of live migration within nodes of a NUMA host.
At a first glance I thought it was enough to periodically checkpoint all VMs and possibly do some cleanup after a node is detected dead to avoid using it again but I’m surely oversimplifying the problem.

Best regards,
Mario