Re: another performance-related thread

[Josh Durgin <josh.durgin@inktank.com>]

On 07/31/2012 08:03 AM, Andrey Korolyov wrote:
> Hi,
>
> I`ve finally managed to run rbd-related test on relatively powerful
> machines and what I have got:
>
> 1) Reads on almost fair balanced cluster(eight nodes) did very well,
> utilizing almost all disk and bandwidth (dual gbit 802.3ad nics, sata
> disks beyond lsi sas 2108 with wt cache gave me ~1.6Gbyte/s on linear
> and sequential reads, which is close to overall disk throughput)
> 2) Writes get much worse, both on rados bench and on fio test when I
> ran fio simularly on 120 vms - at it best, overall performance is
> about 400Mbyte/s, using rados bench -t 12 on three host nodes

How are your osd journals configured? What's your ceph.conf for the
osds?

> fio config:
>
> rw=(randread|randwrite|seqread|seqwrite)
> size=256m
> direct=1
> directory=/test
> numjobs=1
> iodepth=12
> group_reporting
> name=random-ead-direct
> bs=1M
> loops=12
>
> for 120 vm set, Mbyte/s
> linear reads:
> MEAN: 14156
> STDEV: 612.596
> random reads:
> MEAN: 14128
> STDEV: 911.789
> linear writes:
> MEAN: 2956
> STDEV: 283.165
> random writes:
> MEAN: 2986
> STDEV: 361.311
>
> each node holds 15 vms and for 64M rbd cache all possible three states
> - wb, wt and no-cache has almost same numbers at the tests. I wonder
> if it possible to raise write/read ratio somehow. Seems that osd
> underutilize itself, e.g. I am not able to get single-threaded rbd
> write to get above 35Mb/s. Adding second osd on same disk only raising
> iowait time, but not benchmark results.

Are these write tests using direct I/O? That will bypass the cache for
writes, which would explain the similar numbers with different cache
modes.