Re: [PATCH for-4.5 v2] libxc: don't leak buffer containing the uncompressed PV kernel

[Ian Campbell <Ian.Campbell@citrix.com>]

On Thu, 2014-11-20 at 22:13 -0500, Gedalya wrote:
> On 11/20/2014 03:21 PM, Konrad Rzeszutek Wilk wrote:
> > On Thu, Nov 20, 2014 at 03:48:47PM +0000, Ian Campbell wrote:
> >> The libxc xc_dom_* infrastructure uses a very simple malloc memory pool which
> >> is freed by xc_dom_release. However the various xc_try_*_decode routines (other
> >> than the gzip one) just use plain malloc/realloc and therefore the buffer ends
> >> up leaked.
> >>
> >> The memory pool currently supports mmap'd buffers as well as a directly
> >> allocated buffers, however the try decode routines make use of realloc and do
> >> not fit well into this model. Introduce a concept of an external memory block
> >> to the memory pool and provide an interface to register such memory.
> >>
> >> The mmap_ptr and mmap_len fields of the memblock tracking struct lose their
> >> mmap_ prefix since they are now also used for external memory blocks.
> >>
> >> We are only seeing this now because the gzip decoder doesn't leak and it's only
> >> relatively recently that kernels in the wild have switched to better
> >> compression.
> >>
> >> This is https://bugs.debian.org/767295
> >>
> >> Reported by: Gedalya <gedalya@gedalya.net>
> > Gedelya,
> >
> > Could you also test this patch to make sure it does fix the
> > reported issue please?
> 
> So here's what happens now.
> 1. Starts up tiny
> 2. reboot: leak
> 3. reboot: freed (process larger, but the delta is all/mostly shared pages)
> 4. reboot: leak
> 5. reboot: freed
> etc..

WTF, how very strange!

> root@xen:~/xen-pkgs# xl cr /etc/xen/auto/asterisk_deb80.cfg
> Parsing config from /etc/xen/auto/asterisk_deb80.cfg
> root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
> root     22981  0.0  0.0  95968   588 ?        SLsl 21:55   0:00 /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> root@xen:~/xen-pkgs# pmap -x 22981
> 22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> Address           Kbytes     RSS   Dirty Mode  Mapping
> 0000000000400000     144     128       0 r-x-- xl
> 0000000000623000       4       4       4 r---- xl
> 0000000000624000       8       8       8 rw--- xl
> 0000000000626000       4       4       4 rw---   [ anon ]
> 00000000009a6000     288     240     240 rw---   [ anon ]
> 00007f14d4000000     132       8       8 rw---   [ anon ]
> 00007f14d4021000   65404       0       0 -----   [ anon ]
> << snip >>
> ---------------- ------- ------- -------
> total kB           95968    2728     596
> 
> --- reboot domu ---
> 
> root@xen:~/xen-pkgs# ps aux | grep asterisk_deb80
> root     22981  0.6  3.3 131652 20008 ?        SLsl 21:55   0:00 /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> root@xen:~/xen-pkgs# pmap -x 22981
> 22981:   /usr/lib/xen-4.4/bin/xl cr /etc/xen/auto/asterisk_deb80.cfg
> Address           Kbytes     RSS   Dirty Mode  Mapping
> 0000000000400000     144     144       0 r-x-- xl
> 0000000000623000       4       4       4 r---- xl
> 0000000000624000       8       8       8 rw--- xl
> 0000000000626000       4       4       4 rw---   [ anon ]
> 00000000009a6000     288     288     288 rw---   [ anon ]
> 00000000009ee000   35676   16772   16772 rw---   [ anon ]

This is the (temporarily) leaked mapping, right?

> Tried valgrind, it doesn't look like it was able to see what was going on

Indeed. The values for total heap usage at exist and still reachable etc
also don't seem to account for the ~3M of mapping on each iteration.

I don't know how glibc's allocator works, but I suppose it isn't
impossible that it is retaining some mappings of free regions and
collecting them to free later somehow, which just happens to only
trigger every other reboot (e.g. perhaps it is based on some threshold
of free memory).

...investigates...

So, http://man7.org/linux/man-pages/man3/malloc.3.html talks about
special behaviour using mmap for allocations above MMAP_THRESHOLD (128K
by default), which we will be hitting here I think. That explains the
anon mapping.

http://man7.org/linux/man-pages/man3/mallopt.3.html also talks about
various dynamic thresholds for growing and shrinking the heap. My guess
is that we are bouncing up and down over some threshold with every other
reboot.

Ian.