Re: The mfn of the frame, that holds a mlock-ed PV domU usermode page, can change

[Jeremy Fitzhardinge <jeremy@goop.org>]

On 04/12/2010 11:54 AM, Rafal Wojtczuk wrote:
> Hello,
> Would someone on the list enlight me on the following issue, possibly related 
> to mfn management in the PV guest.
> Environment: xen-3.4.3, pvops 2.6.32.9 in dom0 and domU, all 64bit.
> Usermode code
> (if you are interested, at http://gitweb.qubes-os.org/gitweb/?p=mainstream/gui.git;a=blob;f=vchan/vchan/init.c;h=cb2fb851c3b97804b115dbf58fd47a30d6d0a8a3;hb=HEAD)
> in PV domU does the following:
> 1) gets a page via
> ring_ref_v=mmap(0, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE |MAP_ANON,-1, 0);
> 2) mlock(ring_ref_v, 4096)
> 3) determines the mfn number of the frame holding ring_ref_v via a call to 
> u2mfn driver
> (http://gitweb.qubes-os.org/gitweb/?p=mainstream/gui.git;a=blob;f=vchan/u2mfn/u2mfn.c;h=6ff113c07c50ef078ab04d9e61d2faab338357e7;hb=HEAD)
> the driver basically does get_user_pages+kmap+virt_to_mfn (and later
> kunmap+put_page for cleanup).
>   

Yeah, this looks fundamentally suspect.  Using mlock in this way is
going to be fragile.

> Then, the
> usermode code in dom0 does xc_map_foreign_range on the returned mfn, and can
> communicate with the code in domU over this shared page...
> ...but sometimes, apparently the page that backs ring_ref_v changes: if the 
> domU application calls u2mfn ioctl again with ring_ref_v argument, it 
> returns a different mfn.
> And naturally the code in dom0 reads garbage from the address returned by
> its pevious call to xc_map_foreign_range.
>
> I find it puzzling. Is this behaviour normal/expected ?
>   

Yes.

> Mlock man pages say "All pages that contain a part of the specified address 
> range are guaranteed to be resident in RAM", not "be resident at the same
> RAM location", but why would a frame backing a mlock-ed memory be changed ?
> Is there some memory defragmentation going on ?

Yes, the kernel can move usermode memory around to defrag memory.  This
is done to allow higher-order memory allocations to keep working even on
a long-running system which would otherwise fragment the address space. 
Ideally it allows 2M page allocations to succeed.

>  Or maybe only frame->frame
> number function changes (but why would it) ?
>
> Anyway, this behaviour causes problems, as you can see in
> http://www.qubes-os.org/trac/ticket/16#comment:4
> It would be nice if the mfn of a frame that holds a given mlock-ed usermode 
> page could be made constant. 
>
> If you can offer some insight, that would be helpful, particularly:
> 1) Why this does not happen to pages allocated in kernel mode (if it did, it
> would break the split drivers model) ?
>   

No, kernel allocations are not movable by default.

> 2) Can this frame-changing behaviour be switched off at Xen/kernel level?
>   

Not that I know of, and it wouldn't be desirable if it could be.

> 3) Would using grant tables (instead of brutal xc_map_foreign_range) change 
> the situation ?
> BTW, for Qubes it is necessary to map PV domU usermode pages in dom0; 
> particularly, map X server composition buffers.
>   

Why is it necessary to map usermode pages?  It just seems like asking
for trouble.  Why not make it so that the domU X server gets the memory
from the kernel (via some kind of driver), and then map that through to
dom0?

    J