Re: The mfn of the frame, that holds a mlock-ed PV domU usermode page, can change

[Rafal Wojtczuk <rafal@invisiblethingslab.com>]

On Mon, Apr 12, 2010 at 01:39:43PM -0700, Jeremy Fitzhardinge wrote:
> But I assume you have other code which wants to grant through the
> Xorg-allocated framebufer.  That complicates things a bit, but you could
> still add a device (no /proc files, please) with an ioctl which:
> 
>    1. takes a range of usermode addresses
>    2. increments the page refcount for those pages
Or, do not decrement the count, by not calling put_page() ?

>    3. returns the mfns for those pages
> 
> That will prevent the pages from being migrated while you're referring
> to their mfns.
After removing the call to put_page() in u2mfn_ioctl(), see once again
http://gitweb.qubes-os.org/gitweb/?p=mainstream/gui.git;a=blob;f=vchan/u2mfn/u2mfn.c;h=6ff113c07c50ef078ab04d9e61d2faab338357e7;hb=HEAD#l35
the page's mfn changed again.
Even commenting out the kunmap() call in this function did not help, either.
Am I missing something ?

The only working way (for the ring buffer case) is to acquire memory via 
kmalloc and pass it to userspace via remap_pfn_range. But this is unsuitable 
for the case of X composition buffers, because we don't want to alter the
way X allocates memory (it calls plain malloc). We could hijack X's malloc()
via LD_PRELOAD, but then we cannot distinguish which calls are made because
of composition buffer allocation.

>  You need to add something to explicitly decrement the
> refcount to prevent a memory leak, presumably at the time you tear down
> the mapping in dom0.  Ideally you'd arrange to do that triggered off
> unmap of the memory range (by isolating the pages in their own new vma)
> so that it all gets cleaned up on process exit.
By "triggered off unmap" do you mean setting the vm_ops field in struct 
vm_area_struct to a custom struct vm_operations_struct (particularly, with a 
custom close() method), or is there something simpler ?

> I'm not at all familiar with how X manages composition buffers, but it
> seems to me that in normal use, one would want to be able to either
> allocate that buffer in texture memory (so it can be used as a texture
> source), or at least copy updates into texture memory.  Couldn't you
> hook into that transfer to the composition hardware (ie, dom0)?
We are talking about X running in domU; there is no related hardware.
We can determine where the composition buffer is only after it has
been allocated. 

> No, kernel allocations are not movable by default.
Could you mention a few details more on the related migration mechanism ?
E.g. which PG_ flag (set by kmalloc) makes a page unmovable ? Preferably, 
with pointers to relevant code ? 
I guess it is in linux/mm/migrate.c, but I am getting lost
trying to figure out which parts are NUMA specific and which are not; and
particularly, what triggers the migration.

Interestingly, Xorg guys claim X server does nothing special with the memory
acquired by malloc() for the composition buffer. Yet, so far no corruption
of the displayed images have been observed. Maybe a single page vma (that
stores the ring buffer) is particularly attractive for the
migration/defragmentation algorithm, and that is why it is easy to trigger
its relocation (but not so with the composition buffer case) ? 

Once again thanks a lot for your explanations.

Regards,
Rafal Wojtczuk
The Qubes OS Project
http://qubes-os.org