Re: [Qemu-devel] Qemu and heavily increased RSS usage

[Peter Lieven <pl@kamp.de>]

Am 21.06.2016 um 15:18 schrieb Dr. David Alan Gilbert:
> * Peter Lieven (pl@kamp.de) wrote:
>> Hi,
>>
>> while upgrading from Qemu 2.2.0 to Qemu 2.5.1.1 I noticed that the RSS memory usage has heavily increased.
>> We use hugepages so the RSS memory does not include VM memory. In Qemu 2.2.0 it used to be ~30MB per vServer
>> and increased to up to 300 - 400MB for Qemu 2.5.1.1 (same with master). The memory increases over time, but seems
>> not to grow indefinetly. I tried to bisect, but had no result so far that made sense. I also tried valgrind / massif, but
>> valgrind does not see the allocation (at least at exit) and massif fails to rund due to - so it pretends - heap corruption.
>>
>> Any help or ideas how to debug further would be appreciated.
> I think I'd try stripping devices off; can you get a similar difference
> to happen with a guest with no USB, no hugepages, no VGA and a simple
> locally stored IDE disk?

Will do. VGA I already ruled out. Hugepages I can try, but its easier
to monitor the RSS size if the vServer memory is outside the RSS.

>
> If you're having trouble bisecting is it possible it's a change
> in one of the libraries it's linked against?

Same libraries. If I compile qemu-2.2.0 and qemu/master on the exactly
same machine I see the difference.

>
> There was someone asking the other day on #qemu who had a setup that
> was apparently using much more RAM than expected and we didn't
> manage to track it down but I can't remember the version being used.

What I currently suspect are the following changes:
  - We changed the coroutine pool to a per thread model. I disabled the pool. This seems
    to cut the max used RSS to about 150MB which is still a lot more than qemu-2.2.0
  - I suspect that something (e.g. the object based device tree) is creating a lot of small allocations
    which create a massive overhead. I managed to get valgrind/massif running with attached debugger
    and took snapshots of the running VM. What I see is that kernel RSS size is much, much bigger than
    what massif sees as allocated memory. I talk of massif sees 5MB heap usage and the RSS size is 50MB or similar.
  - Changing the memory allocator to tcmalloc or jemalloc seems to relax the issue altough its not gone.
  - VGA memory seems to have been moved from VM memory into Heap. But thats a fixed 16MB allocation.

I will try to cut down devices as you proposed.

Peter