[BUG] memory leak in xend, console output

[BUG] memory leak in xend, console output

[Christopher S. Aker]

2.05 stable tools

My xend process is currently at 462M.  To reproduce: start a domain, telnet/xencons
into the console, make some output (find / is a good way) and watch xend's memory
rise.

Can't restart xend, since it refuses to restart if there are domUs running (check
this post)
http://lists.xensource.com/archives/html/xen-devel/2005-04/msg00051.html

Thanks,
-Chris

Re: [BUG] memory leak in xend

[Christopher S. Aker]

> My xend process is currently at 462M.  To reproduce: start a domain, telnet/xencons
> into the console, make some output (find / is a good way) and watch xend's memory
> rise.

Ok, it's not console output.  xend's memory flickers up/down, but ends on the same
number.

I'm trying "watch -n0 xm list" now. So far, it's gone up 700k in the last minute.  It
appears as though every run through "xm list" list adds ~2k

-Chris

Re: [BUG] memory leak in xend

[M.A. Williamson]

> I'm trying "watch -n0 xm list" now. So far, it's gone up 700k in the 
> last minute. It appears as though every run through "xm list" list adds 
> ~2k

I can replicate this too, so it does need looking into. I can't really see 
what could be wrong with the code, though so would appreciate if someone 
else could poke around in there.

Can you post a link to the graph you mentioned in #xen?

Cheers,
Mark

Re: [BUG] memory leak in xend

[Christopher S. Aker]

> > I'm trying "watch -n0 xm list" now. So far, it's gone up 700k in the
> > last minute. It appears as though every run through "xm list" list adds
> > ~2k
>
> I can replicate this too, so it does need looking into. I can't really see
> what could be wrong with the code, though so would appreciate if someone
> else could poke around in there.
>
> Can you post a link to the graph you mentioned in #xen?

http://www.theshore.net/~caker/xen/memory-week.png

The flat spot towards the end is from when I stopped running the above command.
Thanks for looking into this!

-Chris

Re: [BUG] memory leak in xend

[Anthony Liguori]

Christopher S. Aker wrote:

>http://www.theshore.net/~caker/xen/memory-week.png
>
>The flat spot towards the end is from when I stopped running the above command.
>Thanks for looking into this!
>  
>
I found this article while Googling.

http://evanjones.ca/python-memory.html

It specifically calls out Twisted as being known to cause this sort of 
behavior in the Python VM.

For what it's worth, I've been able to reproduce this memory leakage 
also with Xend.

A workaround for current systems could be a cron job that restarts Xend 
every few days.

Regards,
Anthony Liguori

>-Chris
>
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

Re: [BUG] memory leak in xend

[Matt Ayres]

> 
> A workaround for current systems could be a cron job that restarts Xend 
> every few days.
> 


See other posts... xend can't be restarted for many users.

Re: [BUG] memory leak in xend

[Christopher S. Aker]

> A workaround for current systems could be a cron job that restarts Xend
> every few days.

Xend won't restart in my environment.  The work-around that's been posted does allow
xend to be restarted, but not quite.  It loses the naming of domains (which, I
suppose is to be expected) but other nasty things happen that make it not a reliable
solution for me.

-Chris

Re: [BUG] memory leak in xend

[Adam Heath]

On Thu, 7 Apr 2005, Christopher S. Aker wrote:

> > A workaround for current systems could be a cron job that restarts Xend
> > every few days.
>
> Xend won't restart in my environment.  The work-around that's been posted does allow
> xend to be restarted, but not quite.  It loses the naming of domains (which, I
> suppose is to be expected) but other nasty things happen that make it not a reliable
> solution for me.

Note, that this problaby isn't that bad of a problem in the real world.  We
have a xen box with 4g and 20 domains(including dom0).  64g ram for dom0(128m
swap).  It's been up for 84 days, and python only has 15m in use for xend.

My guess is that as long as you don't do a lot with xm/xend, the memory usage
won't grow.

ps: the above box is running 2 tinderboxes constantly in an instance, and
running a busy mailserver, all over nfsroot.  It's been rock solid(except for
nfs issues, which aren't xen related).

Re: [BUG] memory leak in xend

[Christopher S. Aker]

> Note, that this problaby isn't that bad of a problem in the real world.  We
> have a xen box with 4g and 20 domains(including dom0).  64g ram for dom0(128m
> swap).  It's been up for 84 days, and python only has 15m in use for xend.
>
> My guess is that as long as you don't do a lot with xm/xend, the memory usage
> won't grow.

This happens under real-world usage, as well.  One of our management scripts polls
"xm list" every minute to gather statistics and state of the domains.  The box has
been up for 24 hours, and already xend is using > 120M.  I could see other people
doing something similar very easily and I hope that this problem isn't just dismissed
as a corner case.

I guess it's time to look towards the alternative management tools.  Any chance of
making them work with xen-2.0?

Thanks,
-Chris

RE: [BUG] memory leak in xend

[Ian Pratt]

 > > My xend process is currently at 462M.  To reproduce: start 
> a domain, 
> > telnet/xencons into the console, make some output (find / is a good 
> > way) and watch xend's memory rise.
> 
> Ok, it's not console output.  xend's memory flickers up/down, 
> but ends on the same number.
> 
> I'm trying "watch -n0 xm list" now. So far, it's gone up 700k 
> in the last minute.  It appears as though every run through 
> "xm list" list adds ~2k

Since pyhthon is a garbage collected language, this is not unexpected.
It will reclaim the memory only ehen the GC runs.

You can probably set tuning parameters for python's GC if you want to
reclaim the memory earlier. See python.org.

Ian