From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
To: Peter Lieven <pl@kamp.de>
Cc: Paolo Bonzini <pbonzini@redhat.com>,
qemu-devel@nongnu.org, kwolf@redhat.com,
peter.maydell@linaro.org, mst@redhat.com, mreitz@redhat.com,
kraxel@redhat.com
Subject: Re: [Qemu-devel] [PATCH 03/15] coroutine-ucontext: reduce stack size to 64kB
Date: Tue, 28 Jun 2016 15:20:07 +0100 [thread overview]
Message-ID: <20160628142007.GK2243@work-vm> (raw)
In-Reply-To: <577268DC.9080305@kamp.de>
* Peter Lieven (pl@kamp.de) wrote:
> Am 28.06.2016 um 13:35 schrieb Dr. David Alan Gilbert:
> > * Peter Lieven (pl@kamp.de) wrote:
> > > Am 28.06.2016 um 12:57 schrieb Dr. David Alan Gilbert:
> > > > * Paolo Bonzini (pbonzini@redhat.com) wrote:
> > > > > On 28/06/2016 11:01, Peter Lieven wrote:
> > > > > > evaluation with the recently introduced maximum stack size monitoring revealed
> > > > > > that the actual used stack size was never above 4kB so allocating 1MB stack
> > > > > > for each coroutine is a lot of wasted memory. So reduce the stack size to
> > > > > > 64kB which should still give enough head room.
> > > > > If we make the stack this much smaller, there is a non-zero chance of
> > > > > smashing it. You must add a guard page if you do this (actually more
> > > > > than one because QEMU will happily have stack frames as big as 16 KB).
> > > > > The stack counts for RSS but it's not actually allocated memory, so why
> > > > > does it matter?
> > > > I think I'd be interested in seeing the /proc/.../smaps before and after this
> > > > change to see if anything is visible and if we can see the difference
> > > > in rss etc.
> > > Can you advise what in smaps should be especially looked at.
> > >
> > > As for RSS I can report hat the long term usage is significantly lower.
> > > I had the strange observation that when the VM is running for some minutes
> > > the RSS suddenly increases to the whole stack size.
> > You can see the Rss of each mapping; if you knew where your stacks were
> > it would be easy to see if it was the stacks that were Rss and if
> > there was anything else odd about them.
> > If you set hte mapping as growsdown then you can see the area that has a 'gd'
> > in it's VmFlags.
>
> Would you expect to see each 1MB allocation in smaps or is it possible that
> the kernel merges some mappings to bigger ones?
>
> And more importantly if the regions are merged Paolos comment about we
> do not need a guard page would not be true because a coroutine stack could
> grow into annother coroutines stack. Looking at the commit from Linus it
> would also be good to have that guard page not having the gd flag.
Hmm I'm not sure; one for Paolo.
> Some of the regions above 1024kB have an RSS of exactly 4kB * (Size / 1024kB)
> which leads to the assumption that it is a corouine stack where exactly one page
> has been allocated.
>
> I am asking because this is what I e.g. see for a Qemu VM with flags "gd":
However, what that does show is that if you add up all the Rss, it's still
near-enough nothing worth worrying about.
Maybe it looks different in the old world before you mmap'd it, you could
try going back to the g_malloc'd version but printf'ing the
address you get, then comparing that with smaps to see what the malloc'd
world ended up with mapped.
Dave
> cat /proc/5031/smaps | grep -B18 gd
> 7f808aee7000-7f808b9e6000 rw-p 00000000 00:00 0
> Size: 11264 kB
> Rss: 44 kB
> Pss: 44 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 44 kB
> Referenced: 44 kB
> Anonymous: 44 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f808bb01000-7f8090000000 rw-p 00000000 00:00 0
> Size: 70656 kB
> Rss: 276 kB
> Pss: 276 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 276 kB
> Referenced: 276 kB
> Anonymous: 276 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f80940ff000-7f80943fe000 rw-p 00000000 00:00 0
> Size: 3072 kB
> Rss: 12 kB
> Pss: 12 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 12 kB
> Referenced: 12 kB
> Anonymous: 12 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f8095700000-7f80957ff000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f8097301000-7f8097400000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f80974df000-7f80975de000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> 7f809760c000-7f809770b000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f8097901000-7f8097a00000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f8097b01000-7f8097c00000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f8097d01000-7f8097e00000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f8197f01000-7f8198000000 rw-p 00000000 00:00 0
> Size: 1024 kB
> Rss: 4 kB
> Pss: 4 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 4 kB
> Referenced: 4 kB
> Anonymous: 4 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7f81b4001000-7f81b4200000 rw-p 00000000 00:00 0
> Size: 2048 kB
> Rss: 20 kB
> Pss: 20 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 20 kB
> Referenced: 20 kB
> Anonymous: 20 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac sd
> --
> 7ffd337e2000-7ffd33805000 rw-p 00000000 00:00 0 [stack]
> Size: 144 kB
> Rss: 64 kB
> Pss: 64 kB
> Shared_Clean: 0 kB
> Shared_Dirty: 0 kB
> Private_Clean: 0 kB
> Private_Dirty: 64 kB
> Referenced: 64 kB
> Anonymous: 64 kB
> AnonHugePages: 0 kB
> Shared_Hugetlb: 0 kB
> Private_Hugetlb: 0 kB
> Swap: 0 kB
> SwapPss: 0 kB
> KernelPageSize: 4 kB
> MMUPageSize: 4 kB
> Locked: 0 kB
> VmFlags: rd wr mr mw me gd ac
>
> Peter
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
next prev parent reply other threads:[~2016-06-28 14:20 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-06-28 9:01 [Qemu-devel] [PATCH 00/15] optimize Qemu RSS usage Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 01/15] coroutine-ucontext: mmap stack memory Peter Lieven
2016-06-28 10:02 ` Peter Maydell
2016-06-28 10:21 ` Peter Lieven
2016-06-28 11:04 ` Paolo Bonzini
2016-06-28 9:01 ` [Qemu-devel] [PATCH 02/15] coroutine-ucontext: add a switch to monitor maximum stack size Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 03/15] coroutine-ucontext: reduce stack size to 64kB Peter Lieven
2016-06-28 10:54 ` Paolo Bonzini
2016-06-28 10:57 ` Dr. David Alan Gilbert
2016-06-28 11:17 ` Peter Lieven
2016-06-28 11:35 ` Dr. David Alan Gilbert
2016-06-28 12:09 ` Peter Lieven
2016-06-28 14:20 ` Dr. David Alan Gilbert [this message]
2016-06-30 6:34 ` Peter Lieven
2016-06-28 11:13 ` Peter Lieven
2016-06-28 11:26 ` Paolo Bonzini
2016-06-28 9:01 ` [Qemu-devel] [PATCH 04/15] coroutine: add a knob to disable the shared release pool Peter Lieven
2016-06-28 10:41 ` Paolo Bonzini
2016-06-28 10:47 ` Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 05/15] util: add a helper to mmap private anonymous memory Peter Lieven
2016-10-16 2:10 ` Michael S. Tsirkin
2016-10-18 13:50 ` Alex Bennée
2016-06-28 9:01 ` [Qemu-devel] [PATCH 06/15] exec: use mmap for subpages Peter Lieven
2016-06-28 10:48 ` Paolo Bonzini
2016-06-28 9:01 ` [Qemu-devel] [PATCH 07/15] qapi: use mmap for QmpInputVisitor Peter Lieven
2016-06-28 9:29 ` Dr. David Alan Gilbert
2016-06-28 9:39 ` Peter Lieven
2016-06-28 10:10 ` Daniel P. Berrange
2016-06-28 10:17 ` Dr. David Alan Gilbert
2016-06-28 10:21 ` Daniel P. Berrange
2016-06-28 14:10 ` Eric Blake
2016-06-28 11:36 ` Paolo Bonzini
2016-06-28 14:14 ` Eric Blake
2016-06-30 14:12 ` Markus Armbruster
2016-07-04 9:02 ` Paolo Bonzini
2016-07-04 11:18 ` Markus Armbruster
2016-07-04 11:36 ` Peter Lieven
2016-07-04 11:42 ` Paolo Bonzini
2016-06-28 9:01 ` [Qemu-devel] [PATCH 08/15] virtio: use mmap for VirtQueue Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 09/15] loader: use mmap for ROMs Peter Lieven
2016-06-28 10:41 ` Paolo Bonzini
2016-06-28 11:26 ` Peter Lieven
2016-07-04 7:30 ` Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 10/15] vmware_svga: use mmap for scratch pad Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 11/15] qom: use mmap for bigger Objects Peter Lieven
2016-06-28 10:08 ` Daniel P. Berrange
2016-06-28 10:10 ` Peter Maydell
2016-06-28 10:19 ` Peter Lieven
2016-06-28 10:42 ` Paolo Bonzini
2016-06-28 10:49 ` Peter Lieven
2016-06-30 14:15 ` Markus Armbruster
2016-06-28 9:01 ` [Qemu-devel] [PATCH 12/15] util: add a function to realloc mmapped memory Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 13/15] exec: use mmap for PhysPageMap->nodes Peter Lieven
2016-06-28 10:43 ` Paolo Bonzini
2016-06-28 10:48 ` Peter Lieven
2016-07-11 9:31 ` Peter Lieven
2016-07-11 9:44 ` Peter Lieven
2016-07-11 10:37 ` Paolo Bonzini
2016-07-12 14:34 ` Peter Lieven
2016-07-13 10:27 ` Paolo Bonzini
2016-07-14 14:47 ` Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 14/15] vnc-tight: make the encoding palette static Peter Lieven
2016-06-28 11:12 ` Paolo Bonzini
2016-06-28 11:18 ` Peter Lieven
2016-06-28 9:01 ` [Qemu-devel] [PATCH 15/15] vnc: use mmap for VncState Peter Lieven
2016-06-28 11:37 ` [Qemu-devel] [PATCH 00/15] optimize Qemu RSS usage Paolo Bonzini
2016-06-28 12:14 ` Peter Lieven
2016-06-28 12:29 ` Paolo Bonzini
2016-06-28 12:33 ` Peter Lieven
2016-06-28 12:56 ` Paolo Bonzini
2016-06-28 12:56 ` Dr. David Alan Gilbert
2016-06-28 14:43 ` Peter Lieven
2016-06-28 14:52 ` Peter Lieven
2016-10-12 21:18 ` Michael R. Hines
2016-10-18 10:47 ` Peter Lieven
2016-10-19 17:40 ` Michael R. Hines
2016-10-31 22:00 ` Michael R. Hines
2016-11-01 22:02 ` Michael R. Hines
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160628142007.GK2243@work-vm \
--to=dgilbert@redhat.com \
--cc=kraxel@redhat.com \
--cc=kwolf@redhat.com \
--cc=mreitz@redhat.com \
--cc=mst@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=pl@kamp.de \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).