[Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Luiz Capitulino]

Hello,

Reproducer:

1. Start QEMU with balloon and memory hotplug support:

# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio

2. Check balloon size:

(qemu) info balloon
balloon: actual=1024
(qemu) 

3. Hotplug some memory:

(qemu) object_add memory-backend-ram,id=mem1,size=1G
(qemu) device_add pc-dimm,id=dimm1,memdev=mem1

4. This is step is _not_ needed to reproduce the problem,
   but you may need to online memory manually on Linux so
   that it becomes available in the guest

5. Check balloon size again:

(qemu) info balloon
balloon: actual=1024
(qemu) 

BUG: The guest now has 2GB of memory, but the balloon thinks
     the guest has 1GB

One may think that the problem is that the balloon driver is
ignoring hotplugged memory. This is not what's happening. If
you do balloon your guest, there's nothing stopping the
balloon driver in the guest from ballooning hotplugged memory.

The problem is that the balloon device in QEMU needs to know
the current amount of memory available to the guest.

Before memory hotplug this information was easy to obtain: the
current amount of memory available to the guest is the memory the
guest was booted with. This value is stored in the ram_size global
variable in QEMU and this is what the balloon device emulation
code uses today. However, when memory is hotplugged ram_size is
_not_ updated and the balloon device breaks.

I see two possible solutions for this problem:

1. In addition to reading ram_size, the balloon device in QEMU
   could scan pc-dimm devices to account for hotplugged memory.

   This solution was already implemented by zhanghailiang:

    http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html

   It works, except that on Linux memory hotplug is a two-step
   procedure: first memory is inserted then it has to be onlined
   from user-space. So, if memory is inserted but not onlined
   this solution gives the opposite problem: the balloon device
   will report a larger memory amount than the guest actually has.

   Can we live with that? I guess not, but I'm open for discussion.

   If QEMU could be notified when Linux makes memory online, then
   the problem would be gone. But I guess this can't be done.

2. Modify the balloon driver in the guest to inform the balloon
   device on the host about the current memory available to the
   guest. This way, whenever the balloon device in QEMU needs
   to know the current amount of memory in the guest, it asks
   the guest. This drops any usage of ram_size in the balloon
   device

   I'm not completely sure this is feasible though. For example,
   what happens if the guest reports a memory amount to QEMU and
   right after this more memory is plugged?

   Besides, this solution is more complex than solution 1 and
   won't address older guests.

Another important detail is that, I *suspect* that a very similar
bug already exists with 32-bit guests even without memory
hotplug: what happens if you assign 6GB to a 32-bit without PAE
support? I think the same problem we're seeing with memory
hotplug will happen and solution 1 won't fix this, although
no one seems to care about 32-bit guests...

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[zhanghailiang]

On 2015/2/27 3:26, Luiz Capitulino wrote:
> Hello,
>
> Reproducer:
>
> 1. Start QEMU with balloon and memory hotplug support:
>
> # qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
>
> 2. Check balloon size:
>
> (qemu) info balloon
> balloon: actual=1024
> (qemu)
>
> 3. Hotplug some memory:
>
> (qemu) object_add memory-backend-ram,id=mem1,size=1G
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
>
> 4. This is step is _not_ needed to reproduce the problem,
>     but you may need to online memory manually on Linux so
>     that it becomes available in the guest
>
> 5. Check balloon size again:
>
> (qemu) info balloon
> balloon: actual=1024
> (qemu)
>
> BUG: The guest now has 2GB of memory, but the balloon thinks
>       the guest has 1GB
>
> One may think that the problem is that the balloon driver is
> ignoring hotplugged memory. This is not what's happening. If
> you do balloon your guest, there's nothing stopping the
> balloon driver in the guest from ballooning hotplugged memory.
>
> The problem is that the balloon device in QEMU needs to know
> the current amount of memory available to the guest.
>
> Before memory hotplug this information was easy to obtain: the
> current amount of memory available to the guest is the memory the
> guest was booted with. This value is stored in the ram_size global
> variable in QEMU and this is what the balloon device emulation
> code uses today. However, when memory is hotplugged ram_size is
> _not_ updated and the balloon device breaks.
>
> I see two possible solutions for this problem:
>
> 1. In addition to reading ram_size, the balloon device in QEMU
>     could scan pc-dimm devices to account for hotplugged memory.
>
>     This solution was already implemented by zhanghailiang:
>
>      http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
>
>     It works, except that on Linux memory hotplug is a two-step
>     procedure: first memory is inserted then it has to be onlined
>     from user-space. So, if memory is inserted but not onlined
>     this solution gives the opposite problem: the balloon device
>     will report a larger memory amount than the guest actually has.
>
>     Can we live with that? I guess not, but I'm open for discussion.
>
>     If QEMU could be notified when Linux makes memory online, then
>     the problem would be gone. But I guess this can't be done.
>

Yes, it is really a problem, balloon can't work well with memory block online/offline now.
virtio-balloon can't be notified when memory block online/offline now, actually, we can
add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. (
just a simple register_memory_notifier().)


> 2. Modify the balloon driver in the guest to inform the balloon
>     device on the host about the current memory available to the
>     guest. This way, whenever the balloon device in QEMU needs
>     to know the current amount of memory in the guest, it asks
>     the guest. This drops any usage of ram_size in the balloon
>     device
>
>     I'm not completely sure this is feasible though. For example,
>     what happens if the guest reports a memory amount to QEMU and
>     right after this more memory is plugged?
>

Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon,
why not the memory 'target' size ? Is there any special reason ?

For linux guest, it can always know exactly its current real memory size, but QEMU may not, because
guest can do online/offline memory block by themselves.

If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size
that should  be adjuested. and also can do corresponding action (fill or leak balloon)
when there is online/offline memory block occurred.

>     Besides, this solution is more complex than solution 1 and
>     won't address older guests.
>
> Another important detail is that, I *suspect* that a very similar
> bug already exists with 32-bit guests even without memory
> hotplug: what happens if you assign 6GB to a 32-bit without PAE
> support? I think the same problem we're seeing with memory
> hotplug will happen and solution 1 won't fix this, although
> no one seems to care about 32-bit guests...
>
> .
>

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Luiz Capitulino]

On Fri, 27 Feb 2015 12:09:20 +0800
zhanghailiang <zhang.zhanghailiang@huawei.com> wrote:

> On 2015/2/27 3:26, Luiz Capitulino wrote:
> > Hello,
> >
> > Reproducer:
> >
> > 1. Start QEMU with balloon and memory hotplug support:
> >
> > # qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
> >
> > 2. Check balloon size:
> >
> > (qemu) info balloon
> > balloon: actual=1024
> > (qemu)
> >
> > 3. Hotplug some memory:
> >
> > (qemu) object_add memory-backend-ram,id=mem1,size=1G
> > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> >
> > 4. This is step is _not_ needed to reproduce the problem,
> >     but you may need to online memory manually on Linux so
> >     that it becomes available in the guest
> >
> > 5. Check balloon size again:
> >
> > (qemu) info balloon
> > balloon: actual=1024
> > (qemu)
> >
> > BUG: The guest now has 2GB of memory, but the balloon thinks
> >       the guest has 1GB
> >
> > One may think that the problem is that the balloon driver is
> > ignoring hotplugged memory. This is not what's happening. If
> > you do balloon your guest, there's nothing stopping the
> > balloon driver in the guest from ballooning hotplugged memory.
> >
> > The problem is that the balloon device in QEMU needs to know
> > the current amount of memory available to the guest.
> >
> > Before memory hotplug this information was easy to obtain: the
> > current amount of memory available to the guest is the memory the
> > guest was booted with. This value is stored in the ram_size global
> > variable in QEMU and this is what the balloon device emulation
> > code uses today. However, when memory is hotplugged ram_size is
> > _not_ updated and the balloon device breaks.
> >
> > I see two possible solutions for this problem:
> >
> > 1. In addition to reading ram_size, the balloon device in QEMU
> >     could scan pc-dimm devices to account for hotplugged memory.
> >
> >     This solution was already implemented by zhanghailiang:
> >
> >      http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
> >
> >     It works, except that on Linux memory hotplug is a two-step
> >     procedure: first memory is inserted then it has to be onlined
> >     from user-space. So, if memory is inserted but not onlined
> >     this solution gives the opposite problem: the balloon device
> >     will report a larger memory amount than the guest actually has.
> >
> >     Can we live with that? I guess not, but I'm open for discussion.
> >
> >     If QEMU could be notified when Linux makes memory online, then
> >     the problem would be gone. But I guess this can't be done.
> >
> 
> Yes, it is really a problem, balloon can't work well with memory block online/offline now.
> virtio-balloon can't be notified when memory block online/offline now, actually, we can
> add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. (
> just a simple register_memory_notifier().)

I'm leaning towards applying your series now and do anything that
involves the guest on top if needed.

> > 2. Modify the balloon driver in the guest to inform the balloon
> >     device on the host about the current memory available to the
> >     guest. This way, whenever the balloon device in QEMU needs
> >     to know the current amount of memory in the guest, it asks
> >     the guest. This drops any usage of ram_size in the balloon
> >     device
> >
> >     I'm not completely sure this is feasible though. For example,
> >     what happens if the guest reports a memory amount to QEMU and
> >     right after this more memory is plugged?
> >
> 
> Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon,
> why not the memory 'target' size ? Is there any special reason ?

I don't know either. I guess it's just how the balloon feature was
designed.

> For linux guest, it can always know exactly its current real memory size, but QEMU may not, because
> guest can do online/offline memory block by themselves.
> 
> If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size
> that should  be adjuested. and also can do corresponding action (fill or leak balloon)
> when there is online/offline memory block occurred.

I'm not sure this would work as all sorts of races are possible with
memory allocations that may occur during or after the calculation is
done.

Your series makes the best case work, which is memory is inserted
and the guest uses all of it. Today not even the best case works.

> 
> >     Besides, this solution is more complex than solution 1 and
> >     won't address older guests.
> >
> > Another important detail is that, I *suspect* that a very similar
> > bug already exists with 32-bit guests even without memory
> > hotplug: what happens if you assign 6GB to a 32-bit without PAE
> > support? I think the same problem we're seeing with memory
> > hotplug will happen and solution 1 won't fix this, although
> > no one seems to care about 32-bit guests...
> >
> > .
> >
> 
> 

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Amit Shah]

On (Fri) 27 Feb 2015 [12:09:20], zhanghailiang wrote:
> On 2015/2/27 3:26, Luiz Capitulino wrote:
> >Hello,
> >
> >Reproducer:
> >
> >1. Start QEMU with balloon and memory hotplug support:
> >
> ># qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
> >
> >2. Check balloon size:
> >
> >(qemu) info balloon
> >balloon: actual=1024
> >(qemu)
> >
> >3. Hotplug some memory:
> >
> >(qemu) object_add memory-backend-ram,id=mem1,size=1G
> >(qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> >
> >4. This is step is _not_ needed to reproduce the problem,
> >    but you may need to online memory manually on Linux so
> >    that it becomes available in the guest
> >
> >5. Check balloon size again:
> >
> >(qemu) info balloon
> >balloon: actual=1024
> >(qemu)
> >
> >BUG: The guest now has 2GB of memory, but the balloon thinks
> >      the guest has 1GB
> >
> >One may think that the problem is that the balloon driver is
> >ignoring hotplugged memory. This is not what's happening. If
> >you do balloon your guest, there's nothing stopping the
> >balloon driver in the guest from ballooning hotplugged memory.
> >
> >The problem is that the balloon device in QEMU needs to know
> >the current amount of memory available to the guest.
> >
> >Before memory hotplug this information was easy to obtain: the
> >current amount of memory available to the guest is the memory the
> >guest was booted with. This value is stored in the ram_size global
> >variable in QEMU and this is what the balloon device emulation
> >code uses today. However, when memory is hotplugged ram_size is
> >_not_ updated and the balloon device breaks.
> >
> >I see two possible solutions for this problem:
> >
> >1. In addition to reading ram_size, the balloon device in QEMU
> >    could scan pc-dimm devices to account for hotplugged memory.
> >
> >    This solution was already implemented by zhanghailiang:
> >
> >     http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
> >
> >    It works, except that on Linux memory hotplug is a two-step
> >    procedure: first memory is inserted then it has to be onlined
> >    from user-space. So, if memory is inserted but not onlined
> >    this solution gives the opposite problem: the balloon device
> >    will report a larger memory amount than the guest actually has.
> >
> >    Can we live with that? I guess not, but I'm open for discussion.
> >
> >    If QEMU could be notified when Linux makes memory online, then
> >    the problem would be gone. But I guess this can't be done.
> >
> 
> Yes, it is really a problem, balloon can't work well with memory block online/offline now.
> virtio-balloon can't be notified when memory block online/offline now, actually, we can
> add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. (
> just a simple register_memory_notifier().)

The Linux driver can come to know, but it can't tell the host
out-of-band about it.  A new feature / config option can be added so
that a guest can update the host on what the current available RAM is.

> >2. Modify the balloon driver in the guest to inform the balloon
> >    device on the host about the current memory available to the
> >    guest. This way, whenever the balloon device in QEMU needs
> >    to know the current amount of memory in the guest, it asks
> >    the guest. This drops any usage of ram_size in the balloon
> >    device
> >
> >    I'm not completely sure this is feasible though. For example,
> >    what happens if the guest reports a memory amount to QEMU and
> >    right after this more memory is plugged?
> >
> 
> Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon,
> why not the memory 'target' size ? Is there any special reason ?

This is just how the design / code was done for balloon.  I've
proposed we move to a target-based solution rather than the current
way.  While drafting the new virtio spec, this was considered, but I
lost track of it.  The proposal was to just ditch the current balloon,
and come up with a new one with a saner design.  Don't know who's
keeping track of that, though.

BTW another problem for Luiz's option 2 here is we don't want to wait
for the guest to reply before making decisions.  E.g. the guest could
be in S3 mode, and we may wait indefinitely for a reply, blocking
everything (the situation is slightly better with more threads, but in
older days, blocking for the guest to reply for balloon stats meant
the entire qemu froze till the guest replied.  That's the reason the
feature was disabled).

> For linux guest, it can always know exactly its current real memory size, but QEMU may not, because
> guest can do online/offline memory block by themselves.
> 
> If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size
> that should  be adjuested. and also can do corresponding action (fill or leak balloon)
> when there is online/offline memory block occurred.
> 
> >    Besides, this solution is more complex than solution 1 and
> >    won't address older guests.
> >
> >Another important detail is that, I *suspect* that a very similar
> >bug already exists with 32-bit guests even without memory
> >hotplug: what happens if you assign 6GB to a 32-bit without PAE
> >support? I think the same problem we're seeing with memory
> >hotplug will happen and solution 1 won't fix this, although
> >no one seems to care about 32-bit guests...

Not just 32-bit guests; even 64-bit guests restricted with mem= on the
cmdline.  I know we've discussed this in the past, and I recall
virtio-balloon v2 was going to address this all; sadly I've not kept
uptodate with it.

		Amit

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Luiz Capitulino]

On Mon, 2 Mar 2015 11:52:34 +0530
Amit Shah <amit.shah@redhat.com> wrote:

> > >Another important detail is that, I *suspect* that a very similar
> > >bug already exists with 32-bit guests even without memory
> > >hotplug: what happens if you assign 6GB to a 32-bit without PAE
> > >support? I think the same problem we're seeing with memory
> > >hotplug will happen and solution 1 won't fix this, although
> > >no one seems to care about 32-bit guests...
> 
> Not just 32-bit guests; even 64-bit guests restricted with mem= on the
> cmdline.  

You're right. So, it's an already existing issue that becomes very
apparent with memory hotplug.

> I know we've discussed this in the past, and I recall
> virtio-balloon v2 was going to address this all; sadly I've not kept
> uptodate with it.

Me neither :(

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[zhanghailiang]

On 2015/3/2 14:22, Amit Shah wrote:
> On (Fri) 27 Feb 2015 [12:09:20], zhanghailiang wrote:
>> On 2015/2/27 3:26, Luiz Capitulino wrote:
>>> Hello,
>>>
>>> Reproducer:
>>>
>>> 1. Start QEMU with balloon and memory hotplug support:
>>>
>>> # qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
>>>
>>> 2. Check balloon size:
>>>
>>> (qemu) info balloon
>>> balloon: actual=1024
>>> (qemu)
>>>
>>> 3. Hotplug some memory:
>>>
>>> (qemu) object_add memory-backend-ram,id=mem1,size=1G
>>> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
>>>
>>> 4. This is step is _not_ needed to reproduce the problem,
>>>     but you may need to online memory manually on Linux so
>>>     that it becomes available in the guest
>>>
>>> 5. Check balloon size again:
>>>
>>> (qemu) info balloon
>>> balloon: actual=1024
>>> (qemu)
>>>
>>> BUG: The guest now has 2GB of memory, but the balloon thinks
>>>       the guest has 1GB
>>>
>>> One may think that the problem is that the balloon driver is
>>> ignoring hotplugged memory. This is not what's happening. If
>>> you do balloon your guest, there's nothing stopping the
>>> balloon driver in the guest from ballooning hotplugged memory.
>>>
>>> The problem is that the balloon device in QEMU needs to know
>>> the current amount of memory available to the guest.
>>>
>>> Before memory hotplug this information was easy to obtain: the
>>> current amount of memory available to the guest is the memory the
>>> guest was booted with. This value is stored in the ram_size global
>>> variable in QEMU and this is what the balloon device emulation
>>> code uses today. However, when memory is hotplugged ram_size is
>>> _not_ updated and the balloon device breaks.
>>>
>>> I see two possible solutions for this problem:
>>>
>>> 1. In addition to reading ram_size, the balloon device in QEMU
>>>     could scan pc-dimm devices to account for hotplugged memory.
>>>
>>>     This solution was already implemented by zhanghailiang:
>>>
>>>      http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
>>>
>>>     It works, except that on Linux memory hotplug is a two-step
>>>     procedure: first memory is inserted then it has to be onlined
>>>     from user-space. So, if memory is inserted but not onlined
>>>     this solution gives the opposite problem: the balloon device
>>>     will report a larger memory amount than the guest actually has.
>>>
>>>     Can we live with that? I guess not, but I'm open for discussion.
>>>
>>>     If QEMU could be notified when Linux makes memory online, then
>>>     the problem would be gone. But I guess this can't be done.
>>>
>>
>> Yes, it is really a problem, balloon can't work well with memory block online/offline now.
>> virtio-balloon can't be notified when memory block online/offline now, actually, we can
>> add this capability by using the exist kernel memory hotplug/unplug notifier mechanism. (
>> just a simple register_memory_notifier().)
>
> The Linux driver can come to know, but it can't tell the host
> out-of-band about it.  A new feature / config option can be added so
> that a guest can update the host on what the current available RAM is.
>

This is a feasible scenario. Maybe Luiz could consider this. He is
working on  it now.

>>> 2. Modify the balloon driver in the guest to inform the balloon
>>>     device on the host about the current memory available to the
>>>     guest. This way, whenever the balloon device in QEMU needs
>>>     to know the current amount of memory in the guest, it asks
>>>     the guest. This drops any usage of ram_size in the balloon
>>>     device
>>>
>>>     I'm not completely sure this is feasible though. For example,
>>>     what happens if the guest reports a memory amount to QEMU and
>>>     right after this more memory is plugged?
>>>
>>
>> Hmm, i wonder why we notify the number of pages which should be adjusted to virtio-balloon,
>> why not the memory 'target' size ? Is there any special reason ?
>
> This is just how the design / code was done for balloon.  I've
> proposed we move to a target-based solution rather than the current
> way.  While drafting the new virtio spec, this was considered, but I

OK, it is really a good idea to redesign virtio-balloon which is target-based.

> lost track of it.  The proposal was to just ditch the current balloon,
> and come up with a new one with a saner design.  Don't know who's
> keeping track of that, though.
>

:(

> BTW another problem for Luiz's option 2 here is we don't want to wait
> for the guest to reply before making decisions.  E.g. the guest could
> be in S3 mode, and we may wait indefinitely for a reply, blocking
> everything (the situation is slightly better with more threads, but in
> older days, blocking for the guest to reply for balloon stats meant
> the entire qemu froze till the guest replied.  That's the reason the
> feature was disabled).
>

Got it. thanks.

>> For linux guest, it can always know exactly its current real memory size, but QEMU may not, because
>> guest can do online/offline memory block by themselves.
>>
>> If virtio-balloon in guest know the balloon's 'target' size, it can calculate the exact memory size
>> that should  be adjuested. and also can do corresponding action (fill or leak balloon)
>> when there is online/offline memory block occurred.
>>
>>>     Besides, this solution is more complex than solution 1 and
>>>     won't address older guests.
>>>
>>> Another important detail is that, I *suspect* that a very similar
>>> bug already exists with 32-bit guests even without memory
>>> hotplug: what happens if you assign 6GB to a 32-bit without PAE
>>> support? I think the same problem we're seeing with memory
>>> hotplug will happen and solution 1 won't fix this, although
>>> no one seems to care about 32-bit guests...
>
> Not just 32-bit guests; even 64-bit guests restricted with mem= on the
> cmdline.  I know we've discussed this in the past, and I recall
> virtio-balloon v2 was going to address this all; sadly I've not kept
> uptodate with it.
>
> 		Amit
>
> .
>

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Markus Armbruster]

Luiz Capitulino <lcapitulino@redhat.com> writes:

> Hello,
>
> Reproducer:
>
> 1. Start QEMU with balloon and memory hotplug support:
>
> # qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
>
> 2. Check balloon size:
>
> (qemu) info balloon
> balloon: actual=1024
> (qemu) 
>
> 3. Hotplug some memory:
>
> (qemu) object_add memory-backend-ram,id=mem1,size=1G
> (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
>
> 4. This is step is _not_ needed to reproduce the problem,
>    but you may need to online memory manually on Linux so
>    that it becomes available in the guest
>
> 5. Check balloon size again:
>
> (qemu) info balloon
> balloon: actual=1024
> (qemu) 
>
> BUG: The guest now has 2GB of memory, but the balloon thinks
>      the guest has 1GB

Impact other than "info balloon"?

> One may think that the problem is that the balloon driver is
> ignoring hotplugged memory. This is not what's happening. If
> you do balloon your guest, there's nothing stopping the
> balloon driver in the guest from ballooning hotplugged memory.
>
> The problem is that the balloon device in QEMU needs to know
> the current amount of memory available to the guest.
>
> Before memory hotplug this information was easy to obtain: the
> current amount of memory available to the guest is the memory the
> guest was booted with. This value is stored in the ram_size global
> variable in QEMU and this is what the balloon device emulation
> code uses today. However, when memory is hotplugged ram_size is
> _not_ updated and the balloon device breaks.
>
> I see two possible solutions for this problem:
>
> 1. In addition to reading ram_size, the balloon device in QEMU
>    could scan pc-dimm devices to account for hotplugged memory.
>
>    This solution was already implemented by zhanghailiang:
>
>     http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
>
>    It works, except that on Linux memory hotplug is a two-step
>    procedure: first memory is inserted then it has to be onlined
>    from user-space. So, if memory is inserted but not onlined
>    this solution gives the opposite problem: the balloon device
>    will report a larger memory amount than the guest actually has.
>
>    Can we live with that? I guess not, but I'm open for discussion.
>
>    If QEMU could be notified when Linux makes memory online, then
>    the problem would be gone. But I guess this can't be done.
>
> 2. Modify the balloon driver in the guest to inform the balloon
>    device on the host about the current memory available to the
>    guest. This way, whenever the balloon device in QEMU needs
>    to know the current amount of memory in the guest, it asks
>    the guest. This drops any usage of ram_size in the balloon
>    device

What happens when the guest lies?

>    I'm not completely sure this is feasible though. For example,
>    what happens if the guest reports a memory amount to QEMU and
>    right after this more memory is plugged?
>
>    Besides, this solution is more complex than solution 1 and
>    won't address older guests.
>
> Another important detail is that, I *suspect* that a very similar
> bug already exists with 32-bit guests even without memory
> hotplug: what happens if you assign 6GB to a 32-bit without PAE
> support? I think the same problem we're seeing with memory
> hotplug will happen and solution 1 won't fix this, although
> no one seems to care about 32-bit guests...

Fun...

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Luiz Capitulino]

On Fri, 27 Feb 2015 08:27:00 +0100
Markus Armbruster <armbru@redhat.com> wrote:

> Luiz Capitulino <lcapitulino@redhat.com> writes:
> 
> > Hello,
> >
> > Reproducer:
> >
> > 1. Start QEMU with balloon and memory hotplug support:
> >
> > # qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio
> >
> > 2. Check balloon size:
> >
> > (qemu) info balloon
> > balloon: actual=1024
> > (qemu) 
> >
> > 3. Hotplug some memory:
> >
> > (qemu) object_add memory-backend-ram,id=mem1,size=1G
> > (qemu) device_add pc-dimm,id=dimm1,memdev=mem1
> >
> > 4. This is step is _not_ needed to reproduce the problem,
> >    but you may need to online memory manually on Linux so
> >    that it becomes available in the guest
> >
> > 5. Check balloon size again:
> >
> > (qemu) info balloon
> > balloon: actual=1024
> > (qemu) 
> >
> > BUG: The guest now has 2GB of memory, but the balloon thinks
> >      the guest has 1GB
> 
> Impact other than "info balloon"?

You can only balloon what's reported by "info balloon". If the
guest was booted with 1GB but you hot added another 6GB, then
you'll only be able to balloon 1GB.

> > One may think that the problem is that the balloon driver is
> > ignoring hotplugged memory. This is not what's happening. If
> > you do balloon your guest, there's nothing stopping the
> > balloon driver in the guest from ballooning hotplugged memory.
> >
> > The problem is that the balloon device in QEMU needs to know
> > the current amount of memory available to the guest.
> >
> > Before memory hotplug this information was easy to obtain: the
> > current amount of memory available to the guest is the memory the
> > guest was booted with. This value is stored in the ram_size global
> > variable in QEMU and this is what the balloon device emulation
> > code uses today. However, when memory is hotplugged ram_size is
> > _not_ updated and the balloon device breaks.
> >
> > I see two possible solutions for this problem:
> >
> > 1. In addition to reading ram_size, the balloon device in QEMU
> >    could scan pc-dimm devices to account for hotplugged memory.
> >
> >    This solution was already implemented by zhanghailiang:
> >
> >     http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html
> >
> >    It works, except that on Linux memory hotplug is a two-step
> >    procedure: first memory is inserted then it has to be onlined
> >    from user-space. So, if memory is inserted but not onlined
> >    this solution gives the opposite problem: the balloon device
> >    will report a larger memory amount than the guest actually has.
> >
> >    Can we live with that? I guess not, but I'm open for discussion.
> >
> >    If QEMU could be notified when Linux makes memory online, then
> >    the problem would be gone. But I guess this can't be done.
> >
> > 2. Modify the balloon driver in the guest to inform the balloon
> >    device on the host about the current memory available to the
> >    guest. This way, whenever the balloon device in QEMU needs
> >    to know the current amount of memory in the guest, it asks
> >    the guest. This drops any usage of ram_size in the balloon
> >    device
> 
> What happens when the guest lies?

There are two kinds of guests that would lie: a broken guest or
a malicious guest. For a malicious guest, the worst case I can think
of is that we get the same problem we have today. Not a big deal for
the host, I guess. A broken guest has to be fixed :)

However, I'm getting to the conclusion that this solution will
complicate things even more and may add a bunch of new problems.

I'm leaning towards applying zhanghailiang's series for now. This
series fixes the best case: memory is inserted and the guests uses
all of it right away.

The worst case is: memory is inserted and the guest doesn't use it.
In this case QEMU will allow you to balloon more memory than the
guest is using, which can crash the guest. For example, the guest
is booted with 1GB you hot add 6GB but the guest doesn't use it.
info balloon will report 6GB and will allow you to balloon the guest
down to 2GB, which will crash the guest.

In theory I think this case has always been broken, but in practice
it's very hard (almost impossible?) to reproduce in a Linux 64-bit
guest as you'd have to be able to start the guest with more memory
than it can recognize.

> >    I'm not completely sure this is feasible though. For example,
> >    what happens if the guest reports a memory amount to QEMU and
> >    right after this more memory is plugged?
> >
> >    Besides, this solution is more complex than solution 1 and
> >    won't address older guests.
> >
> > Another important detail is that, I *suspect* that a very similar
> > bug already exists with 32-bit guests even without memory
> > hotplug: what happens if you assign 6GB to a 32-bit without PAE
> > support? I think the same problem we're seeing with memory
> > hotplug will happen and solution 1 won't fix this, although
> > no one seems to care about 32-bit guests...
> 
> Fun...
> 

Re: [Qemu-devel] [BUG] Balloon malfunctions with memory hotplug

[Alexandre DERUMIER]

Hi, 

I think they was already reported some month ago,

and a patch was submitted to the mailing list (but waiting that memory unplug was merged before apply it)

http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html




----- Mail original -----
De: "Luiz Capitulino" <lcapitulino@redhat.com>
À: "qemu-devel" <qemu-devel@nongnu.org>
Cc: "kvm" <kvm@vger.kernel.org>, "Igor Mammedov" <imammedo@redhat.com>, "zhang zhanghailiang" <zhang.zhanghailiang@huawei.com>, pkrempa@redhat.com, "Eric Blake" <eblake@redhat.com>, "Michael S. Tsirkin" <mst@redhat.com>, "amit shah" <amit.shah@redhat.com>
Envoyé: Jeudi 26 Février 2015 20:26:29
Objet: [BUG] Balloon malfunctions with memory hotplug

Hello, 

Reproducer: 

1. Start QEMU with balloon and memory hotplug support: 

# qemu [...] -m 1G,slots=2,maxmem=2G -balloon virtio 

2. Check balloon size: 

(qemu) info balloon 
balloon: actual=1024 
(qemu) 

3. Hotplug some memory: 

(qemu) object_add memory-backend-ram,id=mem1,size=1G 
(qemu) device_add pc-dimm,id=dimm1,memdev=mem1 

4. This is step is _not_ needed to reproduce the problem, 
but you may need to online memory manually on Linux so 
that it becomes available in the guest 

5. Check balloon size again: 

(qemu) info balloon 
balloon: actual=1024 
(qemu) 

BUG: The guest now has 2GB of memory, but the balloon thinks 
the guest has 1GB 

One may think that the problem is that the balloon driver is 
ignoring hotplugged memory. This is not what's happening. If 
you do balloon your guest, there's nothing stopping the 
balloon driver in the guest from ballooning hotplugged memory. 

The problem is that the balloon device in QEMU needs to know 
the current amount of memory available to the guest. 

Before memory hotplug this information was easy to obtain: the 
current amount of memory available to the guest is the memory the 
guest was booted with. This value is stored in the ram_size global 
variable in QEMU and this is what the balloon device emulation 
code uses today. However, when memory is hotplugged ram_size is 
_not_ updated and the balloon device breaks. 

I see two possible solutions for this problem: 

1. In addition to reading ram_size, the balloon device in QEMU 
could scan pc-dimm devices to account for hotplugged memory. 

This solution was already implemented by zhanghailiang: 

http://lists.gnu.org/archive/html/qemu-devel/2014-11/msg02362.html 

It works, except that on Linux memory hotplug is a two-step 
procedure: first memory is inserted then it has to be onlined 
from user-space. So, if memory is inserted but not onlined 
this solution gives the opposite problem: the balloon device 
will report a larger memory amount than the guest actually has. 

Can we live with that? I guess not, but I'm open for discussion. 

If QEMU could be notified when Linux makes memory online, then 
the problem would be gone. But I guess this can't be done. 

2. Modify the balloon driver in the guest to inform the balloon 
device on the host about the current memory available to the 
guest. This way, whenever the balloon device in QEMU needs 
to know the current amount of memory in the guest, it asks 
the guest. This drops any usage of ram_size in the balloon 
device 

I'm not completely sure this is feasible though. For example, 
what happens if the guest reports a memory amount to QEMU and 
right after this more memory is plugged? 

Besides, this solution is more complex than solution 1 and 
won't address older guests. 

Another important detail is that, I *suspect* that a very similar 
bug already exists with 32-bit guests even without memory 
hotplug: what happens if you assign 6GB to a 32-bit without PAE 
support? I think the same problem we're seeing with memory 
hotplug will happen and solution 1 won't fix this, although 
no one seems to care about 32-bit guests... 
-- 
To unsubscribe from this list: send the line "unsubscribe kvm" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html