tmem and swap pagecache

[John Moser <john.r.moser@gmail.com>]

Dragonfly BSD has a feature by which--if you put a swap partition on an 
SSD--it will allow you to configure the swapping area to also accept 
page cache data.  In this way, page cache from spinning hard disks can 
be swapped into SSD for acceleration using the swap device.

It seems to me that creating a tmem[1] module that supplies page cache 
swap or supplies a swap area with page cache logic (i.e. a swap area 
that acts as a combined area, using its entire size as page cache but 
evicting page cache for swap when full) would be a good use of tmem.

It isn't apparent to me that tmem has any way of knowing where a page 
comes from, however.  Specifically, as far as I can tell (I may be 
wrong), you can't hint to tmem that a page is from page cache (you can 
tell it it's persistent or transient--probably swap or cache, but not 
necessarily), much less that it's page cache backed by a spinning disk 
versus an SSD or USB drive.

To my senses, it would be useful to be able to have a database server or 
mass file server with 4GB or 8GB of RAM and a 128GB SSD ($150, cheaper 
than 128GB of RAM...) that can back extended page cache.  Effectively 
you get an L2 page cache, with maybe 2-3GB of L1 page cache in system 
RAM and 108GB of L2 page cache on an SSD, with 20GB in use for the / 
file system (which you want to specifically NOT page cache into SSD).

[Tangent]

It may alternately be a generally good idea to check the backing device 
of page cache in general and favor evicting SSD-backed page cache over 
slow page cache, although now we're getting too specific and we want to 
generically call these devices "fast" or "slow" and more specifically 
"faster than X" and "Slower than Y" if we're discussing that.

Essentially I mean:  what about spinning hard disks on SATA2 vs SATA3 vs 
USB vs eSata?  eSATA is way faster than USB, SATA3 is faster than SATA2 
unless your disk is slower than the SATA2 disk and the SATA2 disk is 
slower than SATA2.  Obviously you want to evict the cache for a 7200RPM 
SATA3 hard disk with 64MB cache more readily than a spinning USB 1.0 
hard disk, if they're both used roughly as recently--the SATA3 drive is 
10% more likely to be used first than the USB drive but the USB drive 
has 500% of the access speed, overall you're likely to come out faster 
favoring the USB drive's page cache.  As you evict more of the SATA3 
drive's cache, what's left has been used far more recently than anything 
on the USB drive, so it starts making more sense to evict the old USB 
drive's data.

[/Tangent]

But even if you favor SSD eviction over hard drive eviction (by any 
metric), you'll eventually come to a point where you only have so much 
page cache and having the ability to swap page cache to a huge SSD 
becomes more attractive.  It becomes an extremely large read cache.  The 
Seagate Momentus XT fares pretty well with an 8GB read cache for this 
(although on their specialized, tightly integrated hardware, they've 
managed to use write-back caching to gain a LOT of write performance, 
too); it would make sense that using an SSD as a big page cache would 
supply similar gains in specialized environments for cheaper than the 
cost of RAM.

Of course that just takes us back to the original question:  Can we hint 
to tmem where the data is coming from, and let it decide if it cares and 
how to handle it?

[1] http://lwn.net/Articles/340409/