Re: [Qemu-devel] Qemu and heavily increased RSS usage

[Peter Lieven <pl@kamp.de>]

Am 24.06.2016 um 11:58 schrieb Peter Maydell:
> On 24 June 2016 at 10:37, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Wed, Jun 22, 2016 at 09:56:06PM +0100, Peter Maydell wrote:
>>> On 22 June 2016 at 20:55, Peter Lieven <pl@kamp.de> wrote:
>>>> What makes the coroutine pool memory intensive is the stack size of 1MB per
>>>> coroutine. Is it really necessary to have such a big stack?
>>> That reminds me that I was wondering if we should allocate
>>> our coroutine stacks with MAP_GROWSDOWN (though if we're
>>> not actually using 1MB of stack then it's only going to
>>> be eating virtual memory, not necessarily real memory.)
>> Yes, MAP_GROWSDOWN will not reduce RSS.
> Right, but then the 1MB of stack as currently allocated isn't
> going to be affecting RSS either I would have thought (except
> transiently, since we zero it on allocation which will
> bring it into the RSS until it falls back out again
> because we don't touch it after that).

What I observe regarding the coroutine pool is really strange. Under I/O load
while booting the vServer the RSS size is low as expected. If the vServer runs
for some time the RSS size suddenly explodes as if suddenly all the stack memory gets
mapped. This symptom definetely goes away if I disable the pool.

Regarding the coroutine pool I had the following thoughts:
 - mmap the stack so its actually really freed if the coroutine is deleted (with MAP_GROWSDOWN or not?)
 - drop the release_pool. It has actually only an effect for non virtio devices where the coroutine is
   not created and deleted in the same thread. But for virtio the release pool has the drawback that there
   is always a pingpong between the release_pool and the alloc_pool.
 - implement some kind of garbage collector that detects that a threads alloc_pool is actually to big (e.g. it
   stays above a watermark for some time) and then reduce its size.
 - detect that a coroutine was created in a vcpu thread (e.g. IDE) and released in the iothread. In this case
   don't add it to the pool so the alloc_pool of the I/O thread does not grow to max without being used.

Peter