[Qemu-devel] [Question] Qemu's Heap Becomes Very Large and Never Reduce Down

[Qemu-devel] [Question] Qemu's Heap Becomes Very Large and Never Reduce Down

[Xulei (Stone)]

Hi, guys

I met a strange problem, with qemu 2.8.1:
qemu consumes too many heap memory after several operations and can not release them anymore:
hot pulg/unplug disk & net, vnc connect/disconnect, guestOS reboot, etc.


01a7a000-3b4efe000 rw-p 00000000 00:00 0                                 [heap]

Size:           15520272 kB

Rss:            14421836 kB

Pss:            14421836 kB

Shared_Clean:          0 kB

Shared_Dirty:          0 kB

Private_Clean:      1164 kB

Private_Dirty:  14420672 kB

Referenced:      7485624 kB

Anonymous:      14421836 kB

AnonHugePages:     34816 kB

Swap:            1098140 kB

KernelPageSize:        4 kB

MMUPageSize:           4 kB

Locked:                0 kB

VmFlags: rd wr mr mw me ac sd

My steps are:
1) start several VMs all equipped only 8G memory;
2) random combining those operations mentioned above;
3) after few hours, qemu's Virt memory and RSS both grow too large and never fall down;

After analysis via /proc/$pid/smaps, I found the VMA of pc.ram does not occupy much
memory but only becauses of heap section.

I guess that has some relations of glibc or qemu rcu_thread, but i can not figure it out.
Is there some patches can fix this problem or does somebody have any idea?
________________________________

Re: [Qemu-devel] [Question] Qemu's Heap Becomes Very Large and Never Reduce Down

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

On Wed, Nov 15, 2017 at 03:14:52AM +0000, Xulei (Stone) wrote:
> Hi, guys
> 
> I met a strange problem, with qemu 2.8.1:
> qemu consumes too many heap memory after several operations and can not release them anymore:
> hot pulg/unplug disk & net, vnc connect/disconnect, guestOS reboot, etc.
> 
> 
> 01a7a000-3b4efe000 rw-p 00000000 00:00 0                                 [heap]
> 
> Size:           15520272 kB
> 
> Rss:            14421836 kB
> 
> Pss:            14421836 kB
> 
> Shared_Clean:          0 kB
> 
> Shared_Dirty:          0 kB
> 
> Private_Clean:      1164 kB
> 
> Private_Dirty:  14420672 kB
> 
> Referenced:      7485624 kB
> 
> Anonymous:      14421836 kB
> 
> AnonHugePages:     34816 kB
> 
> Swap:            1098140 kB
> 
> KernelPageSize:        4 kB
> 
> MMUPageSize:           4 kB
> 
> Locked:                0 kB
> 
> VmFlags: rd wr mr mw me ac sd
> 
> My steps are:
> 1) start several VMs all equipped only 8G memory;
> 2) random combining those operations mentioned above;
> 3) after few hours, qemu's Virt memory and RSS both grow too large and never fall down;
> 
> After analysis via /proc/$pid/smaps, I found the VMA of pc.ram does not occupy much
> memory but only becauses of heap section.
> 
> I guess that has some relations of glibc or qemu rcu_thread, but i can not figure it out.
> Is there some patches can fix this problem or does somebody have any idea?

Please try qemu.git/master.

The malloc implementation (glibc, tcmalloc, jemalloc) probably makes a
difference since you are seeing heap growth.

The main question your report raises is that "random combining those
operations mentioned above" makes it hard to identify the operation that
leads to heap growth.  Can you run isolated tests that do only
hotplug/unplug *or* VNC connect/disconnect *or* guest OS reboot, not
everything together?

Thanks,
Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [Question] Qemu's Heap Becomes Very Large and Never Reduce Down

[Paolo Bonzini]

On 15/11/2017 04:14, Xulei (Stone) wrote:
> Hi, guys
> 
> I met a strange problem, with qemu 2.8.1:
> qemu consumes too many heap memory after several operations and can not
> release them anymore: 
> hot pulg/unplug disk & net, vnc connect/disconnect, guestOS reboot, etc.

Try with newer QEMU; until recently we used a lot of memory at startup,
but it was improved in 2.11.

However, I think Anthony also had a patch adding a malloc_trim call in
the RCU thread, which improved memory usage.  Anthony, do you know if
this is still necessary in 2.11?

Thanks,

Paolo

Re: [Qemu-devel] [Question] Qemu's Heap Becomes Very Large and Never Reduce Down

[Zhong Yang]

On Wed, Nov 15, 2017 at 02:18:55PM +0100, Paolo Bonzini wrote:
> On 15/11/2017 04:14, Xulei (Stone) wrote:
> > Hi, guys
> > 
> > I met a strange problem, with qemu 2.8.1:
> > qemu consumes too many heap memory after several operations and can not
> > release them anymore: 
> > hot pulg/unplug disk & net, vnc connect/disconnect, guestOS reboot, etc.
> 
> Try with newer QEMU; until recently we used a lot of memory at startup,
> but it was improved in 2.11.
> 
> However, I think Anthony also had a patch adding a malloc_trim call in
> the RCU thread, which improved memory usage.  Anthony, do you know if
> this is still necessary in 2.11?
>
  Hello Paolo,

  Last time, when you traveled in BeiJing, we talked about this malloc_trim() 
  solution in RH office. At that time, you asked me to send this patch to upstream.
  I can try this patch in the newest Qemu version if you regard this solution is
  suitable.

  I also want to look for other more better solution, but there are some urgency 
  tasks interrupt me.

  In fact, Anthony also have some other solutions for heap memory deduction in
  our intel ClearContainer.
 
  Regards,

  Yang Zhong
 
> Thanks,
> 
> Paolo

Re: [Qemu-devel] [Question] Qemu's Heap Becomes Very Large and Never Reduce Down

[Xulei (Stone)]

> On Wed, Nov 15, 2017 at 02:18:55PM +0100, Paolo Bonzini wrote:
> > On 15/11/2017 04:14, Xulei (Stone) wrote:
> > > Hi, guys
> > >
> > > I met a strange problem, with qemu 2.8.1:
> > > qemu consumes too many heap memory after several operations and can
> > > not release them anymore:
> > > hot pulg/unplug disk & net, vnc connect/disconnect, guestOS reboot, etc.
> >
> > Try with newer QEMU; until recently we used a lot of memory at
> > startup, but it was improved in 2.11.
> >
> > However, I think Anthony also had a patch adding a malloc_trim call in
> > the RCU thread, which improved memory usage.  Anthony, do you know if
> > this is still necessary in 2.11?
> >
>   Hello Paolo,
> 
>   Last time, when you traveled in BeiJing, we talked about this malloc_trim()
>   solution in RH office. At that time, you asked me to send this patch to
> upstream.
>   I can try this patch in the newest Qemu version if you regard this solution is
>   suitable.
> 
>   I also want to look for other more better solution, but there are some
> urgency
>   tasks interrupt me.
> 
>   In fact, Anthony also have some other solutions for heap memory deduction
> in
>   our intel ClearContainer.
> 

Thanks for reply！
 
I'll test my cases with the upstream qemu ~

In fact, I'd like treat this problem as an unawareness bug. Because:
1) This problem happens NOT during the bootup phase but after several operations.

2) It happens at a low probability, 5 VMs of 10 hosts * 20 VMs (10 hosts, each runs 20 VMs).

3)Once it happens, we can NOT deduct the heap memory through unplugging all disks & nets, 
disconnecting VNC, and rebooting guestOS.

4)Refer to Stefan's advice, I have already isolated my tests that do only one operation persistently 
one time. The heap memory does not grow up very high.

I may think specific operation sequence can trigger this BUG.
I'll do some more tests, and supply more useful information for YOU.

BTW, I am now also following intel ClearContainer and Hyper project, I think they are awesome.

>   Regards,
> 
>   Yang Zhong
> 
> > Thanks,
> >
> > Paolo