Re: Frontswap [PATCH 0/4] (was Transcendent Memory): overview

[Avi Kivity <avi@redhat.com>]

On 05/01/2010 08:10 PM, Dan Magenheimer wrote:
>> Eventually you'll have to swap frontswap pages, or kill uncooperative
>> guests.  At which point all of the simplicity is gone.
>>      
> OK, now I think I see the crux of the disagreement.
>    

Alas, I think we're pretty far from that.

> NO!  Frontswap on Xen+tmem never *never* _never_ NEVER results
> in host swapping.

That's a bug.  You're giving the guest memory without the means to take 
it back.  The result is that you have to _undercommit_ your memory 
resources.

Consider a machine running a guest, with most of its memory free.  You 
give the memory via frontswap to the guest.  The guest happily swaps to 
frontswap, and uses the freed memory for something unswappable, like 
mlock()ed memory or hugetlbfs.

Now the second node dies and you need memory to migrate your guests 
into.  But you can't, and the hypervisor is at the mercy of the guest 
for getting its memory back; and the guest can't do it (at least not 
quickly).

> Host swapping is evil.  Host swapping is
> the root of most of the bad reputation that memory overcommit
> has gotten from VMware customers.  Host swapping can't be
> avoided with some memory overcommit technologies (such as page
> sharing), but frontswap on Xen+tmem CAN and DOES avoid it.
>    

In this case the guest expects that swapped out memory will be slow 
(since was freed via the swap API; it will be slow if the host happened 
to run out of tmem).  So by storing this memory on disk you aren't 
reducing performance beyond what you promised to the guest.

Swapping guest RAM will indeed cause a performance hit, but sometimes 
you need to do it.

> So, to summarize:
>
> 1) You agreed that a synchronous interface for frontswap makes
>     sense for swap-to-in-kernel-compressed-RAM because it is
>     truly swapping to RAM.
>    

Because the interface is internal to the kernel.

> 2) You have pointed out that an asynchronous interface for
>     frontswap makes more sense for KVM than a synchronous
>     interface, because KVM does host swapping.

kvm's host swapping is unrelated.  Host swapping swaps guest-owned 
memory; that's not what we want here.  We want to cache guest swap in 
RAM, and that's easily done by having a virtual disk cached in main 
memory.  We're simply presenting a disk with a large write-back cache to 
the guest.

You could just as easily cache a block device in free RAM with Xen.  
Have a tmem domain behave as the backend for your swap device.  Use 
ballooning to force tmem to disk, or to allow more cache when memory is 
free.

Voila: you no longer depend on guests (you depend on the tmem domain, 
but that's part of the host code), you don't need guest modifications, 
so it works across a wider range of guests.

>    Then you said
>     if you have an asynchronous interface anyway, the existing
>     swap code works just fine with no changes so frontswap
>     is not needed at all... for KVM.
>    

For any hypervisor which implements virtual disks with write-back cache 
in host memory.

> 3) You have suggested that if Xen were more like KVM and required
>     host-swapping, then Xen doesn't need frontswap either.
>    

Host swapping is not a requirement.

> BUT frontswap on Xen+tmem always truly swaps to RAM.
>    

AND that's a problem because it puts the hypervisor at the mercy of the 
guest.

> So there are two users of frontswap for which the synchronous
> interface makes sense.

I believe there is only one.  See below.

> I believe there may be more in the
> future and you disagree but, as Jeremy said, "a general Linux
> principle is not to overdesign interfaces for hypothetical users,
> only for real needs."  We have demonstrated there is a need
> with at least two users so the debate is only whether the
> number of users is two or more than two.
>
> Frontswap is a very non-invasive patch and is very cleanly
> layered so that if it is not in the presence of either of
> the intended "users", it can be turned off in many different
> ways with zero overhead (CONFIG'ed off) or extremely small overhead
> (frontswap_ops is never set; or frontswap_ops is set but the
> underlying hypervisor doesn't support it so frontswap_poolid
> never gets set).
>    

The problem is not the complexity of the patch itself.  It's the fact 
that it introduces a new external API.  If we refactor swapping, that 
stands in the way.

How much, that's up to the mm maintainers to say.  If it isn't a problem 
for them, fine (but I still think 
swap-to-RAM-without-hypervisor-decommit is a bad idea).

> So... KVM doesn't need it and won't use it.  Do you, Avi, have
> any other objections as to why the frontswap patch shouldn't be
> accepted as is for the users that DO need it and WILL use it?
>    

Even ignoring the problems above (which are really hypervisor problems 
and the guest, which is what we're discussing here, shouldn't care if 
the hypervisor paints itself into an oom), a synchronous single-page DMA 
API is a bad idea.  Look at the Xen network and block code, while they 
eventually do a memory copy for every page they see, they try to batch 
multiple pages into an exit, and make the response asynchronous.

As an example, with a batched API you could save/restore the fpu context 
and use sse for copying the memory, while with a single page API you'd 
probably lost out.  Synchronous DMA, even for emulated hardware, is out 
of place in 2010.

-- 
error compiling committee.c: too many arguments to function