From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50105) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bFNMS-0001Vd-Gg for qemu-devel@nongnu.org; Tue, 21 Jun 2016 11:13:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bFNMP-0003iw-1H for qemu-devel@nongnu.org; Tue, 21 Jun 2016 11:13:08 -0400 Received: from mx-v6.kamp.de ([2a02:248:0:51::16]:43533 helo=mx01.kamp.de) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bFNMO-0003gx-NZ for qemu-devel@nongnu.org; Tue, 21 Jun 2016 11:13:04 -0400 References: <5768F923.7040502@kamp.de> <20160621131815.GB2264@work-vm> From: Peter Lieven Message-ID: <57695979.2020102@kamp.de> Date: Tue, 21 Jun 2016 17:12:57 +0200 MIME-Version: 1.0 In-Reply-To: <20160621131815.GB2264@work-vm> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] Qemu and heavily increased RSS usage List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Dr. David Alan Gilbert" Cc: "qemu-devel@nongnu.org" , Paolo Bonzini Am 21.06.2016 um 15:18 schrieb Dr. David Alan Gilbert: > * Peter Lieven (pl@kamp.de) wrote: >> Hi, >> >> while upgrading from Qemu 2.2.0 to Qemu 2.5.1.1 I noticed that the RSS memory usage has heavily increased. >> We use hugepages so the RSS memory does not include VM memory. In Qemu 2.2.0 it used to be ~30MB per vServer >> and increased to up to 300 - 400MB for Qemu 2.5.1.1 (same with master). The memory increases over time, but seems >> not to grow indefinetly. I tried to bisect, but had no result so far that made sense. I also tried valgrind / massif, but >> valgrind does not see the allocation (at least at exit) and massif fails to rund due to - so it pretends - heap corruption. >> >> Any help or ideas how to debug further would be appreciated. > I think I'd try stripping devices off; can you get a similar difference > to happen with a guest with no USB, no hugepages, no VGA and a simple > locally stored IDE disk? Will do. VGA I already ruled out. Hugepages I can try, but its easier to monitor the RSS size if the vServer memory is outside the RSS. > > If you're having trouble bisecting is it possible it's a change > in one of the libraries it's linked against? Same libraries. If I compile qemu-2.2.0 and qemu/master on the exactly same machine I see the difference. > > There was someone asking the other day on #qemu who had a setup that > was apparently using much more RAM than expected and we didn't > manage to track it down but I can't remember the version being used. What I currently suspect are the following changes: - We changed the coroutine pool to a per thread model. I disabled the pool. This seems to cut the max used RSS to about 150MB which is still a lot more than qemu-2.2.0 - I suspect that something (e.g. the object based device tree) is creating a lot of small allocations which create a massive overhead. I managed to get valgrind/massif running with attached debugger and took snapshots of the running VM. What I see is that kernel RSS size is much, much bigger than what massif sees as allocated memory. I talk of massif sees 5MB heap usage and the RSS size is 50MB or similar. - Changing the memory allocator to tcmalloc or jemalloc seems to relax the issue altough its not gone. - VGA memory seems to have been moved from VM memory into Heap. But thats a fixed 16MB allocation. I will try to cut down devices as you proposed. Peter