Re: [RFC PATCH 0/4] (Take 2): transcendent memory ("tmem") for Linux

Avi Kivity
2009-07-12T17:28:17Z

On 07/12/2009 07:28 PM, Dan Magenheimer wrote:
>> Having no struct pages is also a downside; for example this
>> guest cannot
>> have more than 1GB of anonymous memory without swapping like mad.
>> Swapping to tmem is fast but still a lot slower than having
>> the memory
>> available.
>>
>
> Yes, true. Tmem offers little additional advantage for workloads
> that have a huge variation in working set size that is primarily
> anonymous memory. That larger scale "memory shaping" is left to
> ballooning and hotplug.
>

And this is where the policy problems erupt. When do you balloon in
favor of tmem? which guest do you balloon? do you leave it to the
administrator? there's the host's administrator and the guests'
administrators.

CMM2 solves this neatly by providing information to the host. The host
can pick the least recently used page (or a better algorithm) and evict
it using information from the guest, either dropping it or swapping it.
It also provides information back to the guest when it references an
evicted page: either the guest needs to recreate the page or it just
needs to wait.

>> tmem makes life a lot easier to the hypervisor and to the guest, but
>> also gives up a lot of flexibility. There's a difference
>> between memory
>> and a very fast synchronous backing store.
>>
>
> I don't see that it gives up that flexibility. System adminstrators
> are still free to size their guests properly. Tmem's contribution
> is in environments that are highly dynamic, where the only
> alternative is really sizing memory maximally (and thus wasting
> it for the vast majority of time in which the working set is smaller).
>

I meant that once a page is converted to tmem, there's a limited amount
of things you can do with it compared to normal memory. For example
tmem won't help with a dcache intensive workload.

> I'm certainly open to identifying compromises and layer modifications
> that help meet the needs of both Xen and KVM (and others). For
> example, if we can determine that the basic hook placement for
> precache/preswap (or even just precache for KVM) can be built
> on different underlying layers, that would be great!
>

I'm not sure preswap/precache by itself justifies tmem since it can be
emulated by backing the disk with a cached file. What I'm missing in
tmem is the ability for the hypervisor to take a global view on memory;
instead it's forced to look at memory and tmem separately. That's fine
for Xen since it can't really make any decisions on normal memory
(lacking swap); on the other hand kvm doesn't map well to tmem since
"free memory" is already used by the host pagecache.

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.