From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53752) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bHtsB-0005Ik-54 for qemu-devel@nongnu.org; Tue, 28 Jun 2016 10:20:23 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bHts4-0005Cf-L2 for qemu-devel@nongnu.org; Tue, 28 Jun 2016 10:20:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46427) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bHts4-0005CZ-CJ for qemu-devel@nongnu.org; Tue, 28 Jun 2016 10:20:12 -0400 Date: Tue, 28 Jun 2016 15:20:07 +0100 From: "Dr. David Alan Gilbert" Message-ID: <20160628142007.GK2243@work-vm> References: <1467104499-27517-1-git-send-email-pl@kamp.de> <1467104499-27517-4-git-send-email-pl@kamp.de> <20160628105707.GG2243@work-vm> <57725CB0.7090606@kamp.de> <20160628113501.GH2243@work-vm> <577268DC.9080305@kamp.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <577268DC.9080305@kamp.de> Subject: Re: [Qemu-devel] [PATCH 03/15] coroutine-ucontext: reduce stack size to 64kB List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Lieven Cc: Paolo Bonzini , qemu-devel@nongnu.org, kwolf@redhat.com, peter.maydell@linaro.org, mst@redhat.com, mreitz@redhat.com, kraxel@redhat.com * Peter Lieven (pl@kamp.de) wrote: > Am 28.06.2016 um 13:35 schrieb Dr. David Alan Gilbert: > > * Peter Lieven (pl@kamp.de) wrote: > > > Am 28.06.2016 um 12:57 schrieb Dr. David Alan Gilbert: > > > > * Paolo Bonzini (pbonzini@redhat.com) wrote: > > > > > On 28/06/2016 11:01, Peter Lieven wrote: > > > > > > evaluation with the recently introduced maximum stack size monitoring revealed > > > > > > that the actual used stack size was never above 4kB so allocating 1MB stack > > > > > > for each coroutine is a lot of wasted memory. So reduce the stack size to > > > > > > 64kB which should still give enough head room. > > > > > If we make the stack this much smaller, there is a non-zero chance of > > > > > smashing it. You must add a guard page if you do this (actually more > > > > > than one because QEMU will happily have stack frames as big as 16 KB). > > > > > The stack counts for RSS but it's not actually allocated memory, so why > > > > > does it matter? > > > > I think I'd be interested in seeing the /proc/.../smaps before and after this > > > > change to see if anything is visible and if we can see the difference > > > > in rss etc. > > > Can you advise what in smaps should be especially looked at. > > > > > > As for RSS I can report hat the long term usage is significantly lower. > > > I had the strange observation that when the VM is running for some minutes > > > the RSS suddenly increases to the whole stack size. > > You can see the Rss of each mapping; if you knew where your stacks were > > it would be easy to see if it was the stacks that were Rss and if > > there was anything else odd about them. > > If you set hte mapping as growsdown then you can see the area that has a 'gd' > > in it's VmFlags. > > Would you expect to see each 1MB allocation in smaps or is it possible that > the kernel merges some mappings to bigger ones? > > And more importantly if the regions are merged Paolos comment about we > do not need a guard page would not be true because a coroutine stack could > grow into annother coroutines stack. Looking at the commit from Linus it > would also be good to have that guard page not having the gd flag. Hmm I'm not sure; one for Paolo. > Some of the regions above 1024kB have an RSS of exactly 4kB * (Size / 1024kB) > which leads to the assumption that it is a corouine stack where exactly one page > has been allocated. > > I am asking because this is what I e.g. see for a Qemu VM with flags "gd": However, what that does show is that if you add up all the Rss, it's still near-enough nothing worth worrying about. Maybe it looks different in the old world before you mmap'd it, you could try going back to the g_malloc'd version but printf'ing the address you get, then comparing that with smaps to see what the malloc'd world ended up with mapped. Dave > cat /proc/5031/smaps | grep -B18 gd > 7f808aee7000-7f808b9e6000 rw-p 00000000 00:00 0 > Size: 11264 kB > Rss: 44 kB > Pss: 44 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 44 kB > Referenced: 44 kB > Anonymous: 44 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f808bb01000-7f8090000000 rw-p 00000000 00:00 0 > Size: 70656 kB > Rss: 276 kB > Pss: 276 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 276 kB > Referenced: 276 kB > Anonymous: 276 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f80940ff000-7f80943fe000 rw-p 00000000 00:00 0 > Size: 3072 kB > Rss: 12 kB > Pss: 12 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 12 kB > Referenced: 12 kB > Anonymous: 12 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f8095700000-7f80957ff000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f8097301000-7f8097400000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f80974df000-7f80975de000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > 7f809760c000-7f809770b000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f8097901000-7f8097a00000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f8097b01000-7f8097c00000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f8097d01000-7f8097e00000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f8197f01000-7f8198000000 rw-p 00000000 00:00 0 > Size: 1024 kB > Rss: 4 kB > Pss: 4 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 4 kB > Referenced: 4 kB > Anonymous: 4 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7f81b4001000-7f81b4200000 rw-p 00000000 00:00 0 > Size: 2048 kB > Rss: 20 kB > Pss: 20 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 20 kB > Referenced: 20 kB > Anonymous: 20 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac sd > -- > 7ffd337e2000-7ffd33805000 rw-p 00000000 00:00 0 [stack] > Size: 144 kB > Rss: 64 kB > Pss: 64 kB > Shared_Clean: 0 kB > Shared_Dirty: 0 kB > Private_Clean: 0 kB > Private_Dirty: 64 kB > Referenced: 64 kB > Anonymous: 64 kB > AnonHugePages: 0 kB > Shared_Hugetlb: 0 kB > Private_Hugetlb: 0 kB > Swap: 0 kB > SwapPss: 0 kB > KernelPageSize: 4 kB > MMUPageSize: 4 kB > Locked: 0 kB > VmFlags: rd wr mr mw me gd ac > > Peter > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK