wip-librbd-caching

[Martin Mailand <martin@tuxadero.com>]

Hi,

today I tried the wip-librbd-caching branch. The performance improvement 
is very good particular for small writes.
I tested from within a vm with fio:

rbd_cache_enabled=1

fio -name iops -rw=write -size=10G -iodepth 1 -filename /tmp/bigfile 
-ioengine libaio -direct 1 -bs 4k

I get over 10k iops

With an iodepth 4 I get over 30k iops

In comparison with the rbd_writebackwindow I get around 5k iops with an 
iodepth of 1.

So far the whole cluster is running stable for over 12 hours.

But there is also a downside.
My typical vm are 1Gb in size, the default cache size is 200Mb, which is 
20% more memory usage. Maybe 50Mb or less will be enough?
I am going to test that.

The other point is, that the cache is not KSM enabled, therefore 
identical pages will not be merged, could that be changed, what would be 
the downside?

So maybe we could reduce the memory footprint of the cache, but keep 
it's performance.

-martin