RE: [Xen-devel] Re: [RFC PATCH 0/4] (Take 2): transcendent memory ("tmem") for Linux

Dan Magenheimer
2009-07-08T23:33:45Z

Hi Anthony --

Thanks for the comments.

> I have trouble mapping this to a VMM capable of overcommit
> without just coming back to CMM2.
>
> In CMM2 parlance, ephemeral tmem pools is just normal kernel memory
> marked in the volatile state, no?

They are similar in concept, but a volatile-marked kernel page
is still a kernel page, can be changed by a kernel (or user)
store instruction, and counts as part of the memory used
by the VM. An ephemeral tmem page cannot be directly written
by a kernel (or user) store, can only be read via a "get" (which
may or may not succeed), and doesn't count against the memory
used by the VM (even though it likely contains -- for awhile --
data useful to the VM).

> It seems to me that an architecture built around hinting
> would be more
> robust than having to use separate memory pools for this type
> of memory
> (especially since you are requiring a copy to/from the pool).

Depends on what you mean by robust, I suppose. Once you
understand the basics of tmem, it is very simple and this
is borne out in the low invasiveness of the Linux patch.
Simplicity is another form of robustness.

> For instance, you can mark data DMA'd from disk (perhaps by
> read-ahead)
> as volatile without ever bringing it into the CPU cache.
> With tmem, if
> you wanted to use a tmem pool for all of the page cache, you'd likely
> suffer significant overhead due to copying.

The copy may be expensive on an older machine, but on newer
machines copying a page is relatively inexpensive. On a reasonable
multi-VM-kernbench-like benchmark I'll be presenting at Linux
Symposium next week, the overhead is on the order of 0.01%
for a fairly significant savings in IOs.