Re: xenstored memory leak

[Wei Liu <wei.liu2@citrix.com>]

On Wed, Jul 13, 2016 at 02:21:38PM +0200, Juergen Gross wrote:
> On 06/07/16 09:31, Juergen Gross wrote:
> > While testing some patches for support of ballooning in Mini-OS by using
> > the xenstore domain I realized that each xl create/destroy pair would
> > increase memory consumption in Mini-OS by about 5kB. Wondering whether
> > this is a xenstore domain only effect I did the same test with xenstored
> > and oxenstored daemons.
> > 
> > xenstored showed the same behavior, the "referenced" size showed by the
> > pmap command grew by about 5kB for each create/destroy pair.
> > 
> > oxenstored seemed to be even worse in the beginning (about 6kB for each
> > pair), but after about 100 create/destroys the value seemed to be
> > rather stable.
> > 
> > Did anyone notice this memory leak before?
> 
> I think I've found the problem:
> 
> qemu as the device model is setting up a xenstore watch for each backend
> type it is supporting. Unfortunately those watches are never removed
> again. This sums up to the observed memory leak.
> 
> I'm not sure how oxenstored is avoiding the problem, may be by testing
> socket connections to be still alive and so detecting qemu has gone.
> OTOH this won't help for oxenstored running in another domain than the
> device model (either due to oxenstore-stubdom, or a driver domain with
> a qemu based device model).
> 

How unfortunate.

My gut feeling is that xenstored shouldn't have the knowledge to
associate a watch with a "process". The concept of a process is only
meaningful to OS, which wouldn't work on cross-domain xenstored setup.
Maybe the OS xenbus driver should reap all watches on behalf the dead
process. This would also avoid a crashed QEMU leaking resources.

And xenstored should have proper quota support so that a domain can't
set up excessive numbers of watches.

> I'll post a qemu patch to remove those watches on exit soon.
> 
> To find the problem I've added some debug aid to xenstored: when
> a special parameter is specified on invocation it will dump its memory
> allocation structure via talloc_report_full() to a file whenever it is
> receiving a SIGUSR1 signal. Anybody interested in this patch?
> 

Wouldn't hurt to post to the list. If we take it we take it, if we don't
it would still be useful for people who want custom debug patch later.

Wei.

> 
> Juergen
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel