First attempt at rocksdb monitor store stress testing

[Mark Nelson <mark.nelson@inktank.com>]

Hi Guys,

So I've been interested lately in leveldb 99th percentile latency (and 
the amount of write amplification we are seeing) with leveldb. Joao 
mentioned he has written a tool called mon-store-stress in 
wip-leveldb-misc to try to provide a means to roughly guess at what's 
happening on the mons under heavy load.  I cherry-picked it over to 
wip-rocksdb and after a couple of hacks was able to get everything built 
and running with some basic tests.  There was little tuning done and I 
don't know how realistic this workload is, so don't assume this means 
anything yet, but some initial results are here:

http://nhm.ceph.com/mon-store-stress/First%20Attempt.pdf

Command that was used to run the tests:

./ceph-test-mon-store-stress --mon-keyvaluedb <leveldb|rocksdb> 
--write-min-size 50K --write-max-size 2M --percent-write 70 
--percent-read 30 --keep-state --test-seed 1406137270 --stop-at 5000 foo

The most interesting bit right now is that rocksdb seems to be hanging 
in the middle of the test (left it running for several hours).  CPU 
usage on one core was quite high during the hang.  Profiling using perf 
with dwarf symbols I see:

-  49.14%  ceph-test-mon-s  ceph-test-mon-store-stress  [.] unsigned int 
rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned int, 
char const*, unsigned long)
    - unsigned int 
rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned int, 
char const*, unsigned long)
         51.70% rocksdb::ReadBlockContents(rocksdb::RandomAccessFile*, 
rocksdb::Footer const&, rocksdb::ReadOptions const&, 
rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::Env*, bool)
         48.30% 
rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&, 
rocksdb::CompressionType, rocksdb::BlockHandle*)

Not sure what's going on yet, may need to try to enable 
logging/debugging in rocksdb.  Thoughts/Suggestions welcome. :)

Mark