Re: [PATCH 10 of 24] libxc: infrastructure for hypercall safe data buffers

[Jeremy Fitzhardinge <jeremy@goop.org>]

 On 09/08/2010 03:23 AM, Ian Jackson wrote:
> Ian and I discussed this extensively on IRC, during which conversation
> I became convinced that mlock() must do what we want.  Having read the
> code in the kernel I'm not not so sure.
>
> The ordinary userspace access functions are all written to cope with
> pagefaults and retry the access.  So userspace addresses are not in
> general valid in kernel mode even if you've called functions to try to
> test them.  It's not clear what mlock prevents; does it prevent NUMA
> page migration ?  If not then I think indeed the page could be made
> not present by one VCPU editing the page tables while another VCPU is
> entering the hypercall, so that the 2nd VCPU will get a spurious
> EFAULT.

As IanC said, the only thing mlock() guarantees is that accessing the
page won't cause a major fault - ie, need to go to disk to satisfy it. 
You can and will get minor faults on mlocked pages, as a result of the
pte being either non-present or RO.  It can be non-present as a result
of page migration (not necessarily NUMA migration, just defragging
kernel memory to make it possible to allocate higher-order pages), and
RO when doing page-dirtiness tracking.  And I think they can happen
concurrently on different vcpus, so you may end up with a hypercall
being able to start reading the memory, but then fail writing back the
results.

I think the only way to do this properly is to do ioctls out of kernel
memory rather than user process memory.  Perhaps the easiest way to do
this is add an mmap operation to privcmd which allocates a set of kernel
pages and maps them into the process memory, which it can then use as
its hypercall buffer.  The alternatives would be to copy the argument
memory into/out of kernel space around the call, or do some ad-hoc
pinning of pages around the call.  But if we can arrange for all
argument memory to come from a particular buffer, then its easier to
just make sure that buffer has the right properties.

> OTOH: there must be other things that work like Xen - what about user
> mode device drivers of various kinds ?  Do X servers not mlock memory
> and expect to be able to tell the video card to DMA to it ?  etc.
> I think if linux-kernel think that people haven't assumed that mlock()
> actually pins the page, they're mistaken - and it's likely to be not
> just us.

Not really - nothing much depends on keeping a page physically resident
and having a pte in a specific state.  DMA just cares about physical
residency, and you can't do usermode DMA without some way of also
getting the physical address of the page, which would mean you've
already got some kind of kernel driver.  And there would be no way to
make such DMA safe anyway (mlock wouldn't protect against a process
being killed, for example).

Trying to share memory via virtual addresses with an entity which is
entirely external to the kernel is just plain weird.

    J