Re: [RFC] Virtualizing tagged disaggregated memory capacity (app specific, multi host shared)

Gregory Price
2024-09-20T09:06:24Z

> > 2. Coarse grained memory increases for 'normal' memory.
> > Can use memory hot-plug. Recovery of capacity likely to only be possible on
> > VM shutdown.
>
> Is there are reason "movable" (ZONE_MOVABLE) is not an option, at least in
> some setups? If not, why?
>

This seems like a bit of a muddied conversation.

"'normal' memory" has no defined meaning - so lets clear this up a bit

There is:
* System-RAM (memory managed by kernel allocators)
* Special Purpose Memory (generally presented as DAX)

System-RAM is managed as zones - the relevant ones are
* ZONE_NORMAL allows both movable and non-movable allocations
* ZONE_MOVABLE only allows non-movable allocations
(Caveat: this generally only applies to allocation, you can
violate this with stuff like pinning)

Hotplug can be thought of as two discrete mechanisms
* Exposing capacity to the kernel (CXL DCD Transactions)
* Exposing capacity to allocators (mm/memory-hotplug.c)

1) if the intent is to primarily utilize dynamic capacity for VMs, then
the host does not need (read: should not need) to map the memory as
System-RAM in the host. The VMM should be made to consume it directly
via DAX or otherwise.

That capacity is almost by definition "Capital G Guaranteed" to be
reclaimable regardless of what the guest does. A VMM can force a guest
to let go of resources - that's its job.

2) if the intent is to provide dynamic capacity to a host as System-RAM, then
recoverability is dictated by system usage of that capacity. If onlined
into ZONE_MOVABLE, then if the system has avoided doing things like pinning
those pages it should *generally* be recoverable (but not guaranteed).

For the virtualization discussion:

Hotplug and recoverability is a non-issue. The capacity should never be
exposed to system allocators and the VMM should be made to consume special
purpose memory directly. That's on the VMM/orchestration software to get right.

For the host System-RAM discussion:

Auto-onlined hotplug capacity presently defaults to ZONE_NORMAL, but we
discussed (yesterday, at Plumbers) changing this default to ZONE_MOVABLE.

The only concern is when insufficient ZONE_NORMAL exists to support
ZONE_MOVABLE capacity - but this is unlikely to be the general scenario AND
can be mitigated w/ existing mechanisms.

Manually onlined capacity defaults to ZONE_MOVABLE.

It would be nice to make this behavior consistent, since the general opinion
appears to be that this capacity should default to ZONE_MOVABLE.

~Gregory