Re: First attempt at rocksdb monitor store stress testing

[Mark Nelson <mark.nelson@inktank.com>]

Earlier today I modified the rocksdb options so I could enable universal 
compaction.  Over all performance is lower but I don't see the 
hang/stall in the middle of the test either.  Instead the disk is 
basically pegged with 100% writes.  I suspect average latency is higher 
than leveldb, but the highest latency is about 5-6s while we were seeing 
30s spikes for leveldb with levelled (heh) compaction.

I haven't done much tuning either way yet.  It may be that if we keep 
level 0 and level 1 roughly the same size we can reduce stalls in the 
levelled setups.  It will also be interesting to see what happens in 
these tests on SSDs.

Mark

On 07/24/2014 06:13 AM, Mark Nelson wrote:
> Hi Xinxin,
>
> Thanks! I wonder as well if it might be interesting to expose the
> options related to universal compaction?  It looks like rocksdb provides
> a lot of interesting knobs you can adjust!
>
> Mark
>
> On 07/24/2014 12:08 AM, Shu, Xinxin wrote:
>> Hi mark,
>>
>> I think this maybe related to 'verify_checksums' config option ,when
>> ReadOptions is initialized, default this option is  true , all data
>> read from underlying storage will be verified against corresponding
>> checksums,  however,  this option cannot be configured in wip-rocksdb
>> branch. I will modify code to make this option configurable .
>>
>> Cheers,
>> xinxin
>>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org
>> [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Mark Nelson
>> Sent: Thursday, July 24, 2014 7:14 AM
>> To: ceph-devel@vger.kernel.org
>> Subject: First attempt at rocksdb monitor store stress testing
>>
>> Hi Guys,
>>
>> So I've been interested lately in leveldb 99th percentile latency (and
>> the amount of write amplification we are seeing) with leveldb. Joao
>> mentioned he has written a tool called mon-store-stress in
>> wip-leveldb-misc to try to provide a means to roughly guess at what's
>> happening on the mons under heavy load.  I cherry-picked it over to
>> wip-rocksdb and after a couple of hacks was able to get everything
>> built and running with some basic tests.  There was little tuning done
>> and I don't know how realistic this workload is, so don't assume this
>> means anything yet, but some initial results are here:
>>
>> http://nhm.ceph.com/mon-store-stress/First%20Attempt.pdf
>>
>> Command that was used to run the tests:
>>
>> ./ceph-test-mon-store-stress --mon-keyvaluedb <leveldb|rocksdb>
>> --write-min-size 50K --write-max-size 2M --percent-write 70
>> --percent-read 30 --keep-state --test-seed 1406137270 --stop-at 5000 foo
>>
>> The most interesting bit right now is that rocksdb seems to be hanging
>> in the middle of the test (left it running for several hours).  CPU
>> usage on one core was quite high during the hang.  Profiling using
>> perf with dwarf symbols I see:
>>
>> -  49.14%  ceph-test-mon-s  ceph-test-mon-store-stress  [.] unsigned
>> int rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned
>> int, char const*, unsigned long)
>>      - unsigned int
>> rocksdb::crc32c::ExtendImpl<&rocksdb::crc32c::Fast_CRC32>(unsigned
>> int, char const*, unsigned long)
>>           51.70% rocksdb::ReadBlockContents(rocksdb::RandomAccessFile*,
>> rocksdb::Footer const&, rocksdb::ReadOptions const&,
>> rocksdb::BlockHandle const&, rocksdb::BlockContents*, rocksdb::Env*,
>> bool)
>>           48.30%
>> rocksdb::BlockBasedTableBuilder::WriteRawBlock(rocksdb::Slice const&,
>> rocksdb::CompressionType, rocksdb::BlockHandle*)
>>
>> Not sure what's going on yet, may need to try to enable
>> logging/debugging in rocksdb.  Thoughts/Suggestions welcome. :)
>>
>> Mark
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
>