Re: RBD vs RADOS benchmark performance

[Greg <itooo-xVucS5mfmt0AvxtiuMwx3w@public.gmane.org>]

Le 13/05/2013 07:38, Olivier Bonvalet a écrit :
> Le vendredi 10 mai 2013 à 19:16 +0200, Greg a écrit :
>> Hello folks,
>>
>> I'm in the process of testing CEPH and RBD, I have set up a small
>> cluster of  hosts running each a MON and an OSD with both journal and
>> data on the same SSD (ok this is stupid but this is simple to verify the
>> disks are not the bottleneck for 1 client). All nodes are connected on a
>> 1Gb network (no dedicated network for OSDs, shame on me :).
>>
>> Summary : the RBD performance is poor compared to benchmark
>>
>> A 5 seconds seq read benchmark shows something like this :
>>>     sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>>>       0       0         0         0         0 0         -         0
>>>       1      16        39        23   91.9586        92 0.966117  0.431249
>>>       2      16        64        48   95.9602       100 0.513435   0.53849
>>>       3      16        90        74   98.6317       104 0.25631   0.55494
>>>       4      11        95        84   83.9735        40 1.80038   0.58712
>>>   Total time run:        4.165747
>>> Total reads made:     95
>>> Read size:            4194304
>>> Bandwidth (MB/sec):    91.220
>>>
>>> Average Latency:       0.678901
>>> Max latency:           1.80038
>>> Min latency:           0.104719
>> 91MB read performance, quite good !
>>
>> Now the RBD performance :
>>> root@client:~# dd if=/dev/rbd1 of=/dev/null bs=4M count=100
>>> 100+0 records in
>>> 100+0 records out
>>> 419430400 bytes (419 MB) copied, 13.0568 s, 32.1 MB/s
>> There is a 3x performance factor (same for write: ~60M benchmark, ~20M
>> dd on block device)
>>
>> The network is ok, the CPU is also ok on all OSDs.
>> CEPH is Bobtail 0.56.4, linux is 3.8.1 arm (vanilla release + some
>> patches for the SoC being used)
>>
>> Can you show me the starting point for digging into this ?
> You should try to increase read_ahead to 512K instead of the defaults
> 128K (/sys/block/*/queue/read_ahead_kb). I have seen a huge difference
> on reads with that.
>
Olivier,

thanks a lot for pointing this out, it indeed makes a *huge* difference !
> # dd if=/mnt/t/1 of=/dev/zero bs=4M count=100
> 100+0 records in
> 100+0 records out
> 419430400 bytes (419 MB) copied, 5.12768 s, 81.8 MB/s
(caches dropped before each test of course)

Mark, this is probably something you will want to investigate and 
explain in a "tweaking" topic of the documentation.

Regards,
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com