xenheap pages mapped by dom0

xenheap pages mapped by dom0

[George Dunlap]

What's the standard practice regarding freeing xenheap pages that have
been shared with dom0 (and may currently be mapped)?  Do we need to
make sure they're unmapped before we free them, or can we just call
free_xenheap_pages() and let the reference counting take care of
actually freeing them when dom0 unmaps them (if ever)?

I'm looking at allowing xentrace buffers to be resized, and need to
deal with potentially freeing pages which may be mapped by dom0.  If
it's as simple as free-and-forget, I'll include that functionality; if
it's more complicated, I'll punt until another time (as I really only
care about increasing the size of each cpu's buffer, not resizing
buffers).

 -George

Re: xenheap pages mapped by dom0

[Jan Beulich]

>>> George Dunlap <George.Dunlap@eu.citrix.com> 18.12.09 15:04 >>>
>What's the standard practice regarding freeing xenheap pages that have
>been shared with dom0 (and may currently be mapped)?  Do we need to
>make sure they're unmapped before we free them, or can we just call
>free_xenheap_pages() and let the reference counting take care of
>actually freeing them when dom0 unmaps them (if ever)?

No, free_xenheap_pages() is what would be called when the refcount
drops to zero. What you need to do is, afaics, a simply
put_page_and_type(), which undoes the one type and one general
reference share_xen_page_with_guest() installed. This will then
either free the page right away (if Dom0 doesn't have it mapped) or
defer freeing until when Dom0 unmaps it.

Jan

Re: xenheap pages mapped by dom0

[Keir Fraser]

On 18/12/2009 14:29, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> George Dunlap <George.Dunlap@eu.citrix.com> 18.12.09 15:04 >>>
>> What's the standard practice regarding freeing xenheap pages that have
>> been shared with dom0 (and may currently be mapped)?  Do we need to
>> make sure they're unmapped before we free them, or can we just call
>> free_xenheap_pages() and let the reference counting take care of
>> actually freeing them when dom0 unmaps them (if ever)?
> 
> No, free_xenheap_pages() is what would be called when the refcount
> drops to zero. What you need to do is, afaics, a simply
> put_page_and_type(), which undoes the one type and one general
> reference share_xen_page_with_guest() installed. This will then
> either free the page right away (if Dom0 doesn't have it mapped) or
> defer freeing until when Dom0 unmaps it.

In fact not. put_page() calls free_domheap_page() when the refcnt falls to
zero. And free_domheap_page() does not actually free Xen-heap pages: it is
expected that this will happen explicitly later (e.g., when domain is
destroyed) -- this is to deal with Xen itself possibly still using those
pages, and those uses are not reference counted.

I think the right thing to do would be very roughly:
On allocation:
 page = alloc_xenheap_page();
 share_xen_page_with_guest(page);
On deallocation:
 if (test_and_clear(PGC_allocated)) put_page();
 if (page->count_info & PGC_count_mask) return -EBUSY;
 free_xenheap_page(page);

 -- Keir

Re: xenheap pages mapped by dom0

[George Dunlap]

Keir Fraser wrote:
> I think the right thing to do would be very roughly:
> On allocation:
>  page = alloc_xenheap_page();
>  share_xen_page_with_guest(page);
> On deallocation:
>  if (test_and_clear(PGC_allocated)) put_page();
>  if (page->count_info & PGC_count_mask) return -EBUSY;
>  free_xenheap_page(page);
>   
Hmm, the first two should probably happen in the opposite order, no?  
I.e., we should check to see that for each page we want to deallocate, 
page->count_info & PGC_count_mask == 0, returning -EBUSY if that fails; 
then unmark allocate and call free xenheap page?

 -George

Re: xenheap pages mapped by dom0

[Keir Fraser]

On 18/12/2009 15:29, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:

> Hmm, the first two should probably happen in the opposite order, no?
> I.e., we should check to see that for each page we want to deallocate,
> page->count_info & PGC_count_mask == 0, returning -EBUSY if that fails;
> then unmark allocate and call free xenheap page?

No, PGC_allocated holds a reference count. Reference count cannot be zero
while PGC_allocated is true.

 -- Keir

Re: xenheap pages mapped by dom0

[Jan Beulich]

>>> Keir Fraser <keir.fraser@eu.citrix.com> 18.12.09 16:24 >>>
>I think the right thing to do would be very roughly:
>On allocation:
> page = alloc_xenheap_page();
> share_xen_page_with_guest(page);
>On deallocation:
> if (test_and_clear(PGC_allocated)) put_page();
> if (page->count_info & PGC_count_mask) return -EBUSY;

Hmm, and how would you recover from that? I don't think George wants
his enclosing operation to fail because of a page still being mapped. On
x86-64, at least, it might be possible to play with PGC_xen_heap, to
convert a Xen heap page to a domain heap one, but that wouldn't
cover anyone else (and would seem rather hackish, if it works at all).

> free_xenheap_page(page);

Jan

Re: xenheap pages mapped by dom0

[Keir Fraser]

On 18/12/2009 15:44, "Jan Beulich" <JBeulich@novell.com> wrote:

>> On deallocation:
>> if (test_and_clear(PGC_allocated)) put_page();
>> if (page->count_info & PGC_count_mask) return -EBUSY;
> 
> Hmm, and how would you recover from that? I don't think George wants
> his enclosing operation to fail because of a page still being mapped. On
> x86-64, at least, it might be possible to play with PGC_xen_heap, to
> convert a Xen heap page to a domain heap one, but that wouldn't
> cover anyone else (and would seem rather hackish, if it works at all).

I assume he would wrap it all up in a retryable hypercall. E.g., I don't
know the scenario we're thinking about, but perhaps xentrace tools go bad
and leak mappings, so the hypercall fails. So you restart xentrace tools,
and now the hypercall would work on next attempt. You wouldn't have to be
stuck forever, but yes you are stuck, or at least have to keep tracking that
page in Xen, until dom0 stops mapping the page, obviously!

 -- Keir

Re: xenheap pages mapped by dom0

[George Dunlap]

Jan Beulich wrote:
>>>> Keir Fraser <keir.fraser@eu.citrix.com> 18.12.09 16:24 >>>
>>>>         
>> I think the right thing to do would be very roughly:
>> On allocation:
>> page = alloc_xenheap_page();
>> share_xen_page_with_guest(page);
>> On deallocation:
>> if (test_and_clear(PGC_allocated)) put_page();
>> if (page->count_info & PGC_count_mask) return -EBUSY;
>>     
>
> Hmm, and how would you recover from that? I don't think George wants
> his enclosing operation to fail because of a page still being mapped. On
> x86-64, at least, it might be possible to play with PGC_xen_heap, to
> convert a Xen heap page to a domain heap one, but that wouldn't
> cover anyone else (and would seem rather hackish, if it works at all).
>   
I wouldn't really mind the op failing; I just want it to fail cleanly. 
All-or-nothing would be optimal IMHO.  Failing such that everything will 
automatically clean itself up if/when dom0 does unmap the pages would be 
OK too.  Having half the pages freed / marked unallocated, but some 
still allocated doesn't seem very good.

Anyway, I care more about allocating larger buffers than I do about 
re-allocating, so maybe I'll just punt on freeing maybe-mapped buffers 
for now.

However, that brings up another point: suppose that I do several 
allocations, one per cpu, and one fails.  Now I want to go and free the 
buffers I've just allocated (which shouldn't be mapped in dom0 yet).   
Can I not just call free_xenheap_pages()? Do I need to clear 
PGC_allocated before calling?

Is any of this stuff written down somewhere? :-)

 -George

Re: xenheap pages mapped by dom0

[Jan Beulich]

>>> George Dunlap <george.dunlap@eu.citrix.com> 18.12.09 17:19 >>>
>However, that brings up another point: suppose that I do several 
>allocations, one per cpu, and one fails.  Now I want to go and free the 
>buffers I've just allocated (which shouldn't be mapped in dom0 yet).   
>Can I not just call free_xenheap_pages()? Do I need to clear 
>PGC_allocated before calling?

If you would fully trust Dom0 (including that it will never run malicious
code), then you could do so I think. But security people may not like
that, so it would still seem be better to verify by clearing the allocated
flag, putting the page, and checking its refcount before freeing.

>Is any of this stuff written down somewhere? :-)

I'm afraid it isn't.

Jan

Re: xenheap pages mapped by dom0

[Keir Fraser]

On 18/12/2009 16:19, "George Dunlap" <George.Dunlap@eu.citrix.com> wrote:

> However, that brings up another point: suppose that I do several
> allocations, one per cpu, and one fails.  Now I want to go and free the
> buffers I've just allocated (which shouldn't be mapped in dom0 yet).
> Can I not just call free_xenheap_pages()? Do I need to clear
> PGC_allocated before calling?

Once you have called share_xen_page_with_guest(), dom0 can map that page at
any time and it is not safe to free_xenheap_page() without first checking
for guest mappings.

Perhaps you can lazily adjust tarce buffers downwards in size? I.e. Have a
go, remember if you fail, and try again on future buffer-adjustment
hypercalls?

> Is any of this stuff written down somewhere? :-)

No. :-)

 -- Keir

Re: xenheap pages mapped by dom0

[George Dunlap]

On Fri, Dec 18, 2009 at 4:40 PM, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> Once you have called share_xen_page_with_guest(), dom0 can map that page at
> any time and it is not safe to free_xenheap_page() without first checking
> for guest mappings.
>
> Perhaps you can lazily adjust tarce buffers downwards in size? I.e. Have a
> go, remember if you fail, and try again on future buffer-adjustment
> hypercalls?

I don't expect buffer-adjustment to be a common thing, so the "lazy"
option might end up being incredibly lazy. :-)  My plan now is to do
alloc-once only (i.e., keeping the same semantics), and make all the
allocations before sharing the page with dom0.  In the future, I may
add the tear-down and check for reference count, with an "undo" if
there are still outstanding reference counts.  I think the most likely
reason to run into this scenario is if someone accidentally starts
xentrace or xenmon when there's already an instance of one of them
running, in which case the right thing to do is fail the new one and
let the user decide whether to kill the old process or not.

 -George