Re: [uml-devel] Re: [Lhms-devel] [PATCH 0/7] Fragmentation Avoidance V19

[Rob Landley <rob@landley.net>]

On Friday 04 November 2005 13:10, Blaisorblade wrote:
> > If you've got a daemon running in the virtual system to hand back memory
> > to the host, then you don't need a tuneable.
>
> I think Jeff's idea was a daemon running on the host (not as root) to
> manage splitting of memory between UMLs (and possibly the host).

That's more configuration on the host that's not really needed.  Doesn't do my 
case any good.

> > What I was thinking is that if we get prezeroing infrastructure that can
> > use various prezeroing accelerators (as has been discussed but I don't
> > believe merged), then a logical prezeroing accelerator for UML would be
> > calling madvise on the host system.  This has the advantage of
> > automatically giving back to the host system any memory that's not in
> > use, but would require some way to tell kswapd or some such that keeping
> > around lots of prezeroed memory is preferable to keeping around lots of
> > page cache.
>
> Ah, ok, I see, but a tuneable to say this is almost useless for anything
> else I guess, so it won't even get coded.

If we get prezeroing, the tunable is useful.  If we haven't got prezeroing, 
this infrastructure probably won't get in.

> > I think it's
> > because the disk is so overwhelmed, and some things (like vim's .swp
> > file, and something similar in kmail's composer) do a gratuitous fsync...
>
> Yep, that's possible (running Gentoo, I often go to loads like 8-10,
> including a CPU-hog in the background, and things become a bit slow).

It's not load for me, it's disk bandwidth.  Every time it writes to the swap 
UBD, that data is scheduled for write-out.  So if it's thrashing the swap 
file, even though it's reading the data back in fairly quickly the data still 
gets written out to disk, again and again, each time it's touched.  Result: 
the disk I/O becomes a bottleneck and the disk is _PEGGED_ as long as the 
swap storm continues.

Not try to have anything else on the system that wants to access the disk.  
Yes, reads are prioritized but anything that does fsync and waits for a write 
(as vi and kmail's composer do, or anything _else_ that wants to swap out a 
page to free up some memory) winds up waiting somewhere around 15 seconds.  
(Yeah, 20 gigs/second on linear reads.  Not quite so much on an endless 
series of small random seeks.)

> However, I feel that really it's the simple "fork" which slows down like a
> crawl (and given that memory allocation will easily sleep waiting for some
> memory to be freed - i.e. to be freed or synced to disk, that's
> reasonable).

 You don't have to fork to block in this swap storm.  The fact my old ubuntu's 
on a 2.6.10 kernel might have something to do with it, though...

> And, btw, Frag. Avoidance would help for that too...

If it ever goes in...

> > > However look at /proc/sys/vm/swappiness
> >
> > Setting swappiness to 0 triggers the OOM killer on 2.6.14 for a load that
> > completes with swappiness at 60.
>
> Yep, I see - it becomes so reluctant to swapping that it prefers killing.
> Unintended, but at least a reasonable bug...

Triggering the OOM killer when you have _any_ writes pending is silly.  Wait 
and memory will free up.  And yet we do this all the time.  We trigger the 
OOM killer when there's still swap space, too.  I thought the point of a swap 
block device instead of a swap file was that there were no memory allocations 
needed to flush out memory.

The oom killer is theoretically for when waiting won't help.  But the 
implementation doesn't seem to match that...

> > I mentioned this on the list a little
> > while ago and some people asked for copies of my test script...
> >
> > > or use Con Kolivas's patches to find new tunable and policies.
> >
> > The daemon you mentioned is an alternative, but I'm not quite sure how
> > rapid the daemon's reaction is going to be to potential OOM situations
> > when something suddenly wants an extra 200 megs...
>
> The daemon will have to be designed and written, so we'll see... and we
> _could_ add a pre-OOM hook (it would be meaningful for Xen and any other
> virtualization tool)... to trigger a mconsole notification on the host and
> wait for any response from the daemon...
>
> At that point I become curious for "how much should the daemon give to the
> guest", and that would be policy configurable... but the policy file (which
> I already guess will be more complex than the daemon itself) would like
> some way to gather "how memory it needs" informations.
>
> We already started discussing on IRC with Jeff some ideas for estimating
> the past usage, but predicting the future one is more difficult.
>
> It's still possible to calculate the speed of new allocations, but not to
> now what's happening inside... the only possibility I see is to allow the
> notification to include the amount of needed memory (you can already do
> "echo something nice > /proc/notify", we now only need a client).
>
> But this allows DoSing the host with untrusted users. Not fully though,
> since you can never hotplug memory which wasn't hot-unplugged first - i.e.
> you would boot your UML with mem=256m and then immediately hot-unplug the
> most of it.

In theory, UML memory isn't all that much different from allocating normal 
user memory.  So a DOS shouldn't enter into it.

I like the idea that a UML instance can figure out when it has extra memory 
it's not using, and hand it back to the host.  And for a UML to maintain any 
significant quantity of page cache that isn't tmpfs or ramfs is probably a 
bad idea (with the possible exception of NFS mounts).  With hostfs or UBD, 
the host should have it cached to it's just two copies of memory, and 
fetching it again is easy.  (Dentries I can see cacheing lots of.)

Hence a UML instance could indeed (in theory) have lots of actually free 
memory (aggressively reclaim page cache) that it could madvise back to the 
host.  And if it can do that, why does it need the deamon?

Rob


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel