Having trouble with ballooning

Having trouble with ballooning

[Moritz Duge]

Hi,
I had some trouble while using the ballooning feature of KVM (using 
Ubuntu 10.04 with standard software versions).


The first scenario:
1. Having a guest started by this command: qemu -enable-kvm -m 768 
-balloon virtio -cdrom linux_2.6.34.iso
The guest is running Linux 2.6.34 including the ballooning driver.
2. Entering "balloon 256" and a few seconds later "info balloon" in the 
Qemu monitor. Qemu will report, the guest uses 256mb of memory now. The 
guest is reporting the same (using "free" for example).
3. Entering "change ide1-cd0 linux_2.6.18.iso" to change the guests 
CD-ROM to another image, containing a Linux kernel without ballooning 
driver.
4. Rebooting the guest.
5. After booting the 2.6.18-OS, it will report it has 768mb memory 
(using "free). But Qemu monitor will still tell 256, when entering "info 
memory".
I know why this happens. But is this a good behaviour? Shouldn't Qemu 
tell something like "maybe 256, but there is no more balloon driver in 
the guest and maybe it uses the full 768 now"???


The second scenario:
After the first scenario, the guest can also really start using the 
additional 512mb of memory (768 - 256)!!! I think this shouldn't happen 
or at least there should be an option to allow or deny this. Or at least 
least least this should be printed in big letters in the man-pages or 
somewhere else where everyone will read it!
Because before I experienced this, I assumed I can be sure the guest 
can't get back the memory which was freed using ballooning. So if I use 
the memory freed by ballooning for some other qemu-instances and the 
first one starts to use those memory again, all Qemu instances will 
crash (this is what actually happens in most cases).
What I'm asking for, is a way to force the guest to stay in the memory I 
assigned by ballooning. And if the guest tries to use more memory (maybe 
because it just unloaded the ballooning driver) the guest should crash, 
but the host shouldn't get in any trouble!!!
This can be really annoying. I think a very common use-case for 
virtualization is, to run untrusted software or unsecure webservices in 
a vm, so the bad software can't do anything to the host or other VMs on 
the host. But when using ballooning, the bad software can! It's no 
"remote code execution", but the guest can consume a lot of memory and 
cause the host or at least the other VMs on the host to crash.


The third scenario:
1. Booting a machine with a guest not having a ballooning driver. (e.g. 
qemu -enable-kvm -m 768 -balloon virtio -cdrom linux_2.6.18.iso)
2. Adjusting the memory by "balloon 512" in the qemu monitor.
3. Qemu won't report that it couldn't adjust the memory. Instead it will 
wait until the guest loads a ballooning driver. Is this a good 
behaviour? Shouldn't there be at least a switch in the qemu monitor for 
the command "balloon". So if I use the switch when changing the memory 
(e.g. "balloon -h 256"), qemu won't try to change the memory later and 
it will tell me "error: no ballooning driver found".


Thanks for reading and thanks a lot, if there will be a solution for 
this, specially for scenario two.

Greetings
Moritz Duge

-- 
Artfiles New Media GmbH | Heidenkampsweg 100 | 20097 Hamburg
Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
E-Mail:support@artfiles.de  | Web:http://www.artfiles.de
Geschäftsführer: Carsten Bals | Harald Oltmanns | Tim Evers
Eingetragen im Handelsregister Hamburg - HRB 81478

Re: Having trouble with ballooning

[Moritz Duge]

Hi there again,
just wanted to ask, why I'm getting no answers? Everyone's busy?
Has this already been discussed (I wouldn't actually believe, if I see 
how Qemu behaves for example in scenario 1)?

If you'd like to have more details about the issue or anything else, 
please contact me!

Thanks!
Moritz Duge



05.08.2010 13:41 by Moritz Duge:
> Hi,
> I had some trouble while using the ballooning feature of KVM (using 
> Ubuntu 10.04 with standard software versions).
>
>
> The first scenario:
> 1. Having a guest started by this command: qemu -enable-kvm -m 768 
> -balloon virtio -cdrom linux_2.6.34.iso
> The guest is running Linux 2.6.34 including the ballooning driver.
> 2. Entering "balloon 256" and a few seconds later "info balloon" in 
> the Qemu monitor. Qemu will report, the guest uses 256mb of memory 
> now. The guest is reporting the same (using "free" for example).
> 3. Entering "change ide1-cd0 linux_2.6.18.iso" to change the guests 
> CD-ROM to another image, containing a Linux kernel without ballooning 
> driver.
> 4. Rebooting the guest.
> 5. After booting the 2.6.18-OS, it will report it has 768mb memory 
> (using "free). But Qemu monitor will still tell 256, when entering 
> "info memory".
> I know why this happens. But is this a good behaviour? Shouldn't Qemu 
> tell something like "maybe 256, but there is no more balloon driver in 
> the guest and maybe it uses the full 768 now"???
>
>
> The second scenario:
> After the first scenario, the guest can also really start using the 
> additional 512mb of memory (768 - 256)!!! I think this shouldn't 
> happen or at least there should be an option to allow or deny this. Or 
> at least least least this should be printed in big letters in the 
> man-pages or somewhere else where everyone will read it!
> Because before I experienced this, I assumed I can be sure the guest 
> can't get back the memory which was freed using ballooning. So if I 
> use the memory freed by ballooning for some other qemu-instances and 
> the first one starts to use those memory again, all Qemu instances 
> will crash (this is what actually happens in most cases).
> What I'm asking for, is a way to force the guest to stay in the memory 
> I assigned by ballooning. And if the guest tries to use more memory 
> (maybe because it just unloaded the ballooning driver) the guest 
> should crash, but the host shouldn't get in any trouble!!!
> This can be really annoying. I think a very common use-case for 
> virtualization is, to run untrusted software or unsecure webservices 
> in a vm, so the bad software can't do anything to the host or other 
> VMs on the host. But when using ballooning, the bad software can! It's 
> no "remote code execution", but the guest can consume a lot of memory 
> and cause the host or at least the other VMs on the host to crash.
>
>
> The third scenario:
> 1. Booting a machine with a guest not having a ballooning driver. 
> (e.g. qemu -enable-kvm -m 768 -balloon virtio -cdrom linux_2.6.18.iso)
> 2. Adjusting the memory by "balloon 512" in the qemu monitor.
> 3. Qemu won't report that it couldn't adjust the memory. Instead it 
> will wait until the guest loads a ballooning driver. Is this a good 
> behaviour? Shouldn't there be at least a switch in the qemu monitor 
> for the command "balloon". So if I use the switch when changing the 
> memory (e.g. "balloon -h 256"), qemu won't try to change the memory 
> later and it will tell me "error: no ballooning driver found".
>
>
> Thanks for reading and thanks a lot, if there will be a solution for 
> this, specially for scenario two.
>
> Greetings
> Moritz Duge

Re: Having trouble with ballooning

[Marcelo Tosatti]

On Thu, Aug 05, 2010 at 01:41:16PM +0200, Moritz Duge wrote:
> Hi,
> I had some trouble while using the ballooning feature of KVM (using
> Ubuntu 10.04 with standard software versions).
> 
> 
> The first scenario:
> 1. Having a guest started by this command: qemu -enable-kvm -m 768
> -balloon virtio -cdrom linux_2.6.34.iso
> The guest is running Linux 2.6.34 including the ballooning driver.
> 2. Entering "balloon 256" and a few seconds later "info balloon" in
> the Qemu monitor. Qemu will report, the guest uses 256mb of memory
> now. The guest is reporting the same (using "free" for example).
> 3. Entering "change ide1-cd0 linux_2.6.18.iso" to change the guests
> CD-ROM to another image, containing a Linux kernel without
> ballooning driver.
> 4. Rebooting the guest.
> 5. After booting the 2.6.18-OS, it will report it has 768mb memory
> (using "free). But Qemu monitor will still tell 256, when entering
> "info memory".
> I know why this happens. But is this a good behaviour? Shouldn't
> Qemu tell something like "maybe 256, but there is no more balloon
> driver in the guest and maybe it uses the full 768 now"???

What version of qemu-kvm are you using? Reporting should work 
properly with a recent qemu-kvm version.

> The second scenario:
> After the first scenario, the guest can also really start using the
> additional 512mb of memory (768 - 256)!!! I think this shouldn't
> happen or at least there should be an option to allow or deny this.
> Or at least least least this should be printed in big letters in the
> man-pages or somewhere else where everyone will read it!
> Because before I experienced this, I assumed I can be sure the guest
> can't get back the memory which was freed using ballooning. So if I
> use the memory freed by ballooning for some other qemu-instances and
> the first one starts to use those memory again, all Qemu instances
> will crash (this is what actually happens in most cases).
> What I'm asking for, is a way to force the guest to stay in the
> memory I assigned by ballooning. And if the guest tries to use more
> memory (maybe because it just unloaded the ballooning driver) the
> guest should crash, but the host shouldn't get in any trouble!!!
> This can be really annoying. I think a very common use-case for
> virtualization is, to run untrusted software or unsecure webservices
> in a vm, so the bad software can't do anything to the host or other
> VMs on the host. But when using ballooning, the bad software can!
> It's no "remote code execution", but the guest can consume a lot of
> memory and cause the host or at least the other VMs on the host to
> crash.

Ballooning requires guest cooperation. If you want to enforce memory
limits, take a look at cgroups (Documentation/cgroups/memory.txt in the
kernel source tree).

> 
> 
> The third scenario:
> 1. Booting a machine with a guest not having a ballooning driver.
> (e.g. qemu -enable-kvm -m 768 -balloon virtio -cdrom
> linux_2.6.18.iso)
> 2. Adjusting the memory by "balloon 512" in the qemu monitor.
> 3. Qemu won't report that it couldn't adjust the memory. Instead it
> will wait until the guest loads a ballooning driver. Is this a good
> behaviour? Shouldn't there be at least a switch in the qemu monitor
> for the command "balloon". So if I use the switch when changing the
> memory (e.g. "balloon -h 256"), qemu won't try to change the memory
> later and it will tell me "error: no ballooning driver found".

You see that that guest has not ballooned down with the output 
from "info balloon".

> 
> Thanks for reading and thanks a lot, if there will be a solution for
> this, specially for scenario two.
> 
> Greetings
> Moritz Duge
> 
> -- 
> Artfiles New Media GmbH | Heidenkampsweg 100 | 20097 Hamburg
> Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
> E-Mail:support@artfiles.de  | Web:http://www.artfiles.de
> Geschäftsführer: Carsten Bals | Harald Oltmanns | Tim Evers
> Eingetragen im Handelsregister Hamburg - HRB 81478
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Having trouble with ballooning

[Moritz Duge]

Am 23.08.2010 20:16, schrieb Marcelo Tosatti:
> On Thu, Aug 05, 2010 at 01:41:16PM +0200, Moritz Duge wrote:
>    
>> Hi,
>> I had some trouble while using the ballooning feature of KVM (using
>> Ubuntu 10.04 with standard software versions).
>>
>>
>> The first scenario:
>> 1. Having a guest started by this command: qemu -enable-kvm -m 768
>> -balloon virtio -cdrom linux_2.6.34.iso
>> The guest is running Linux 2.6.34 including the ballooning driver.
>> 2. Entering "balloon 256" and a few seconds later "info balloon" in
>> the Qemu monitor. Qemu will report, the guest uses 256mb of memory
>> now. The guest is reporting the same (using "free" for example).
>> 3. Entering "change ide1-cd0 linux_2.6.18.iso" to change the guests
>> CD-ROM to another image, containing a Linux kernel without
>> ballooning driver.
>> 4. Rebooting the guest.
>> 5. After booting the 2.6.18-OS, it will report it has 768mb memory
>> (using "free). But Qemu monitor will still tell 256, when entering
>> "info memory".
>> I know why this happens. But is this a good behaviour? Shouldn't
>> Qemu tell something like "maybe 256, but there is no more balloon
>> driver in the guest and maybe it uses the full 768 now"???
>>      
> What version of qemu-kvm are you using? Reporting should work
> properly with a recent qemu-kvm version.
>    
I tested it on 2 systems:
- Ubuntu 10.04 and it's native version of Qemu+KVM (it's a 0.12.3)

- Debian 5.0.5 on which I compiled a 2.6.34 kernel, it's KVM module
   and Qemu 0.12.4 from sources.

On both systems, the command "info balloon" outputs the value I set,
when the ballooning has been still active. And it continues to report
this value, also if the guest has no more ballooning driver.
It also continues to report this value, when the guest starts to use
more, then the ballooned down memory.
(If you have a look at scenario one: If the guests starts to use more
then 256mb, up to 768, "info balloon" still reports 256)

For testing I actually booted a Debian 5.0.5 guest, ballooned down
the memory to 256, inserted a Kubuntu 6.06 cdrom-iso and rebooted
the guest from that cdrom-iso. Kubuntu 6.06 comes with a 2.6.18
kernel, so it has no virtio support at all.
After that, I created a tmpfs-mount in the Kubuntu guest and filled
it using "dd if=/dev/hdc of=/tmpfs-mount/foobar".


>> The second scenario:
>> After the first scenario, the guest can also really start using the
>> additional 512mb of memory (768 - 256)!!! I think this shouldn't
>> happen or at least there should be an option to allow or deny this.
>> Or at least least least this should be printed in big letters in the
>> man-pages or somewhere else where everyone will read it!
>> Because before I experienced this, I assumed I can be sure the guest
>> can't get back the memory which was freed using ballooning. So if I
>> use the memory freed by ballooning for some other qemu-instances and
>> the first one starts to use those memory again, all Qemu instances
>> will crash (this is what actually happens in most cases).
>> What I'm asking for, is a way to force the guest to stay in the
>> memory I assigned by ballooning. And if the guest tries to use more
>> memory (maybe because it just unloaded the ballooning driver) the
>> guest should crash, but the host shouldn't get in any trouble!!!
>> This can be really annoying. I think a very common use-case for
>> virtualization is, to run untrusted software or unsecure webservices
>> in a vm, so the bad software can't do anything to the host or other
>> VMs on the host. But when using ballooning, the bad software can!
>> It's no "remote code execution", but the guest can consume a lot of
>> memory and cause the host or at least the other VMs on the host to
>> crash.
>>      
> Ballooning requires guest cooperation. If you want to enforce memory
> limits, take a look at cgroups (Documentation/cgroups/memory.txt in the
> kernel source tree).
>
>    
Wohooo! That's what I was looking for!
It's just a little sad, that I have to guess the overhead made by the
Qemu process itself, to add it to the memory limit I want to set.
A hard ballooning implemented by Qemu would look a little "cleaner" I
think. But for the beginning this is a nice way to do it, also because
it gives the possibility to use the swap if the guest ignores the
ballooning, so the guest doesn't has to crash.


>>
>> The third scenario:
>> 1. Booting a machine with a guest not having a ballooning driver.
>> (e.g. qemu -enable-kvm -m 768 -balloon virtio -cdrom
>> linux_2.6.18.iso)
>> 2. Adjusting the memory by "balloon 512" in the qemu monitor.
>> 3. Qemu won't report that it couldn't adjust the memory. Instead it
>> will wait until the guest loads a ballooning driver. Is this a good
>> behaviour? Shouldn't there be at least a switch in the qemu monitor
>> for the command "balloon". So if I use the switch when changing the
>> memory (e.g. "balloon -h 256"), qemu won't try to change the memory
>> later and it will tell me "error: no ballooning driver found".
>>      
> You see that that guest has not ballooned down with the output
> from "info balloon".
>
>    
>> Thanks for reading and thanks a lot, if there will be a solution for
>> this, specially for scenario two.
>>
>> Greetings
>> Moritz Duge
>>
>> -- 
>> Artfiles New Media GmbH | Heidenkampsweg 100 | 20097 Hamburg
>> Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
>> E-Mail:support@artfiles.de  | Web:http://www.artfiles.de
>> Geschäftsführer: Carsten Bals | Harald Oltmanns | Tim Evers
>> Eingetragen im Handelsregister Hamburg - HRB 81478
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>      
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
I think the result of the first scenario could be considered as a bug!

About the second scenario:
Please add to the documentation and the part about ballooning in the
man-page, that ballooning is not ensuring the memory limit. This could
save a lot of people from crashing their hosts.
Also make a short hint, that you can use cgroup to enforce memory
limits. I think an url to a cgroup tutorial would be also nice for a
lot of people!

Thanks!
Moritz Duge