Balloon device in qemu?

Balloon device in qemu?

[Avi Kivity]

Marcelo, the balloon was last seen drifting over your territory.  Care 
to brush it up and send it over?

-- 
error compiling committee.c: too many arguments to function

Re: Balloon device in qemu?

[Marcelo Tosatti]

On Thu, Jul 31, 2008 at 04:01:54PM +0300, Avi Kivity wrote:
> Marcelo, the balloon was last seen drifting over your territory.  Care  
> to brush it up and send it over?

Anthony rewrote the backend, I think this is the latest version:

http://www.archivum.info/qemu-devel@nongnu.org/2008-04/msg00080.html

Re: Balloon device in qemu?

[Anthony Liguori]

Marcelo Tosatti wrote:
> On Thu, Jul 31, 2008 at 04:01:54PM +0300, Avi Kivity wrote:
>   
>> Marcelo, the balloon was last seen drifting over your territory.  Care  
>> to brush it up and send it over?
>>     
>
> Anthony rewrote the backend, I think this is the latest version:
>
> http://www.archivum.info/qemu-devel@nongnu.org/2008-04/msg00080.html
>   

Is there a way to detect MMU notifiers from userspace?  I don't think 
it's currently safe to madvise unconditionally.

Regards,

Anthony Liguori

Re: Balloon device in qemu?

[Andrea Arcangeli]

On Fri, Aug 01, 2008 at 02:47:39PM -0500, Anthony Liguori wrote:
> Is there a way to detect MMU notifiers from userspace?  I don't think it's 
> currently safe to madvise unconditionally.

There is no way to detect mmu notifiers from userspace (well strictly
speaking you could check /proc/kallsyms) but the point is that without
mmu notifiers madvise won't be enough (the memory may not be freed
without the other ioctl proposed by Marcelo that also flushes sptes).

Especially with the plan to pin pages during memslot allocation (next
step in the kvm-userland compatibility effort when built against old
kernel) madvise will be a noop because all memory in the memslot will
remain pinned.

The safety issue with madvise was then found to be the same issue for
all rmap_remove invocations (not just the ones done by Marcelo's
ioctl). Whenever the put_page in rmap_remove happens to be the last
free of the page, we have a tlb race where we flush the other vcpus
tlbs after the page was already freed by rmap_remove. This is usually
only an issue for smp guest on smp host.  So in short madvise run on a
memslot wasn't "less" safe, but still the ioctl proposed to remove
sptes to allow memory to be released wasn't capable of fixing this
race for madvise.

mmu notifiers on the other hand are allowing madvise to free all
memory reliably while fixing this race as well as a whole (not just
for madvise but for swapping and all other operations too).

The compatibility plan for the host kernels without mmu notifiers is
going to entirely prevent swapping, and in turn it will never allow
rmap_remove to invoke the last free of the page will also fix the tlb
troubles.

So if you currently just madvise, right now it won't be less safe than
without madvise (it's not safe regardless) but without mmu notifier
madvise won't remove sptes so it won't be reliable (this is why once
Marcelo proposed the ioctl to call after/before madvise, but that
ioctl still had the same issues that every other rmap_remove had
without mmu notifier). With the next plan of pinning all pages you can
still madvise just fine, and it will be safe too (like every other
rmap_remove will be safe) but it will be a guaranteed a noop because
of the memslot being entirely page-pinning. So madvise on old kernels
is never less safe but if we want to provide reliable ballooning to
host kernels without mmu notifiers with the next compatibility plan
that Avi suggested, we should "trim" the memslot first (to unpin the
pages) instead of calling madvise and then you can munmap the range
instead of madvise. Or you can add a new call that only drops sptes in
the range and _later_ unpin the pages (that's similar to Marcelo ioctl
but the unpin event happening later should fix its previous safety
issues). However if the sptes removal is done without touching the
memslot that will complicate the memslots semantics quite a bit
(because all memory mapped by a memslot won't be guaranteed to be
entirely pinned anymore and we'll have to track separately the
balloned ranges that aren't pinned). So I guess teaching
set_memory_region to trim a memslot (currently it can't) sounds one
approach that could be used to allow balloning on kernels without mmu
notifiers, if we go with Avi's backwards compatibility suggestion of
memslot pinning. Then it's up to you if to munamp or madvise, it'll be
the same then.

It's a bit complicated if something isn't clear on what the issues are
without mmu notifier, don't hesitate to ask more details.

Re: Balloon device in qemu?

[Anthony Liguori]

Andrea Arcangeli wrote:
> On Fri, Aug 01, 2008 at 02:47:39PM -0500, Anthony Liguori wrote:
>   
>> Is there a way to detect MMU notifiers from userspace?  I don't think it's 
>> currently safe to madvise unconditionally.
>>     
>
> There is no way to detect mmu notifiers from userspace (well strictly
> speaking you could check /proc/kallsyms) but the point is that without
> mmu notifiers madvise won't be enough (the memory may not be freed
> without the other ioctl proposed by Marcelo that also flushes sptes).
>   

Right.  Ideally, I'd like to just disable the balloon device if mmu 
notifiers *and* kvm support for it are not present.  If you're saying 
there is no way to probe for this now from userspace, then that's 
something we need to add.

Regards,

Anthony Liguori

Re: Balloon device in qemu?

[Gerd Hoffmann]

Anthony Liguori wrote:
> 
> Right.  Ideally, I'd like to just disable the balloon device if mmu
> notifiers *and* kvm support for it are not present.  If you're saying
> there is no way to probe for this now from userspace, then that's
> something we need to add.

I think it is there, IIRC the kvm mmu notifier pull request on lkml had
a few patches from Andrea with all the mm magic and one from avi adding
a feature flag.  Checking ...

#define KVM_CAP_SYNC_MMU 16  /* Changes to host mmap are reflected in
guest */

This one I guess ...

cheers,
  Gerd

-- 
http://kraxel.fedorapeople.org/xenner/

Re: Balloon device in qemu?

[Andrea Arcangeli]

On Mon, Aug 04, 2008 at 10:32:58AM +0200, Gerd Hoffmann wrote:
> #define KVM_CAP_SYNC_MMU 16  /* Changes to host mmap are reflected in
> guest */
> 
> This one I guess ...

That doesn't currently tell if kvm is using mmu notifiers though,
because it's set unconditionally. If we want to use it to tell if kvm
is using mmu notifiers we've to wrap it around an #ifdef or to use the
awk hack, which is trivial to add.

I'm more worried about the fact that with Anthony's plan ballooning
will be unusable for a long while on enterprise distros, as it will
take a while before they run on a >=2.6.27 kernel. So I wonder if a
compact layer is worth it. But surely initially we can modify
KVM_CAP_SYNC_MMU to not be set if kernel isn't compiled with
CONFIG_MMU_NOTIFIER=y and we can add the compact layer later if needed.

Re: Balloon device in qemu?

[Avi Kivity]

Andrea Arcangeli wrote:
> On Mon, Aug 04, 2008 at 10:32:58AM +0200, Gerd Hoffmann wrote:
>   
>> #define KVM_CAP_SYNC_MMU 16  /* Changes to host mmap are reflected in
>> guest */
>>
>> This one I guess ...
>>     
>
> That doesn't currently tell if kvm is using mmu notifiers though,
> because it's set unconditionally. If we want to use it to tell if kvm
> is using mmu notifiers we've to wrap it around an #ifdef or to use the
> awk hack, which is trivial to add.
>
> I'm more worried about the fact that with Anthony's plan ballooning
> will be unusable for a long while on enterprise distros, as it will
> take a while before they run on a >=2.6.27 kernel. So I wonder if a
> compact layer is worth it. But surely initially we can modify
> KVM_CAP_SYNC_MMU to not be set if kernel isn't compiled with
> CONFIG_MMU_NOTIFIER=y and we can add the compact layer later if needed.
>   

I added something to hack-module.awk, so KVM_CAP_SYNC_MMU is now reliable.

-- 
error compiling committee.c: too many arguments to function

Re: Balloon device in qemu?

[Anthony Liguori]

Avi Kivity wrote:
> Andrea Arcangeli wrote:
>> I'm more worried about the fact that with Anthony's plan ballooning
>> will be unusable for a long while on enterprise distros, as it will
>> take a while before they run on a >=2.6.27 kernel. So I wonder if a
>> compact layer is worth it. But surely initially we can modify
>> KVM_CAP_SYNC_MMU to not be set if kernel isn't compiled with
>> CONFIG_MMU_NOTIFIER=y and we can add the compact layer later if needed.
>>   
>
> I added something to hack-module.awk, so KVM_CAP_SYNC_MMU is now 
> reliable.

Great, I'll update the balloon backend and resubmit this afternoon.

Regards,

Anthony Liguori