Re: Proposed new "memory capacity claim" hypercall/feature

[Tim Deegan <tim@xen.org>]

At 10:06 -0700 on 29 Oct (1351505175), Dan Magenheimer wrote:
> Hypervisor design/implementation overview:
> 
> A domain currently does RAM accounting with two primary counters
> "tot_pages" and "max_pages".  (For now, let's ignore shr_pages,
> paged_pages, and xenheap_pages, and I hope Olaf/Andre/others can
> provide further expertise and input.)
> 
> Tot_pages is a struct_domain element in the hypervisor that tracks
> the number of physical RAM pageframes "owned" by the domain.  The
> hypervisor enforces that tot_pages is never allowed to exceed another
> struct_domain element called max_pages.
> 
> I would like to introduce a new counter, which records how
> much capacity is claimed for a domain which may or may not yet be
> mapped to physical RAM pageframes.  To do so, I'd like to split
> the concept of tot_pages into two variables, tot_phys_pages and
> tot_claimed_pages and require the hypervisor to also enforce:
> 
> d.tot_phys_pages <= d.tot_claimed_pages[3] <= d.max_pages
> 
> I'd also split the hypervisor global "total_avail_pages" into
> "total_free_pages" and "total_unclaimed_pages".  (I'm definitely
> going to need to study more the two-dimensional array "avail"...)
> The hypervisor must now do additional accounting to keep track
> of the sum of claims across all domains and also enforce the
> global:
> 
> total_unclaimed_pages <= total_free_pages
> 
> I think the memory_op hypercall can be extended to add two
> additional subops, XENMEM_claim and XENMEM_release.  (Note: To
> support tmem, there will need to be two variations of XEN_claim,
> "hard claim" and "soft claim" [3].)  The XEN_claim subop atomically
> evaluates total_unclaimed_pages against the new claim, claims
> the pages for the domain if possible and returns success or failure.
> The XEN_release "unsets" the domain's tot_claimed_pages (to an
> "illegal" value such as zero or MINUS_ONE).
> 
> The hypervisor must also enforce some semantics:  If an allocation
> occurs such that a domain's tot_phys_pages would equal or exceed
> d.tot_claimed_pages, then d.tot_claimed_pages becomes "unset".
> This enforces the temporary nature of a claim:  Once a domain
> fully "occupies" its claim, the claim silently expires.

Why does that happen?  If I understand you correctly, releasing the
claim is something the toolstack should do once it knows it's no longer
needed.

> In the case of a dying domain, a XENMEM_release operation
> is implied and must be executed by the hypervisor.
> 
> Ideally, the quantity of unclaimed memory for each domain and
> for the system should be query-able.  This may require additional
> memory_op hypercalls.
> 
> I'd very much appreciate feedback on this proposed design!

As I said, I'm not opposed to this, though even after reading through
the other thread I'm not convinced that it's necessary (except in cases
where guest-controlled operations are allowed to consume unbounded
memory, which frankly gives me the heebie-jeebies).

I think it needs a plan for handling restricted memory allocations.
For example, some PV guests need their memory to come below a
certain machine address, or entirely in superpages, and certain
build-time allocations come from xenheap.  How would you handle that
sort of thing?

Cheers,

Tim.