Re: [PATCH] Add hypercall to mark superpages to improve performance

[Keir Fraser <keir.fraser@eu.citrix.com>]

On 30/04/2010 14:43, "Dave McCracken" <dcm@mccr.org> wrote:

>> That could be implemented with no extra hypercalls, and I reckon it's
>> probably easier to make this race-free too. Obviously it does have extra
>> code complexity to construct this array (which I suppose needs to be
>>  sparse, just like page_info array, in the face of very sparse memory
>>  maps). The space overhead (about 8 bytes per 2MB, or 0.0004% of total
>>  system memory) would be trivial. Compared with an extra reference count in
>>  every page_info, which would have a much higher 0.2% overhead.
> 
> I like this idea.  I'll look into it.

The algorithm for acquiring a superpage refcount would be something like:
 y = superpage_info->count
 do {
    x = y
    if ( x == 0 )
        for (each page in super_page)
            if (!get_page(page))
                goto undo_and_fail;
 } while ((y = cmpxchg(&superpage_info->count, x, x+1)) != x);

For destroying a superpage refcount:
 y = superpage_info->count
 do { x = y } while ((y = cmpxchg(..,x,x-1)) != x);
 if (x==1) for (each page in super_page) put_page(page)

I'd actually have two refcounts in superpage struct: one for read-only
mappings and one for read-write mappings. The latter would be updated as
above except for the use of {get,put}_page_and_type() instead of
{get,put}_page().

The other thing to note is that this approach is cheap when updating
superpage refcounts between non-zero values. If regularly
constructing/destroying the *only* superpage mapping of a page, then
obviously you are going to be continually taking the slow path. In that
case, pinning a superpage with new hypercalls as you suggested may have to
be done. It kind of depends on how your workloads interact with the
reference-counting. In any case, you could implement the basic version as
described here, and add hypercalls as a second stage if they turn out to be
needed.

I can help with further details and advice if need be.

 -- Keir