Re: poor write performance

[Mark Nelson <mark.nelson@inktank.com>]

On 04/19/2013 08:30 PM, James Harper wrote:
>>> rados -p <pool> -b 4096 bench 300 seq -t 64
>>
>> sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>>       0       0         0         0         0         0         -         0
>> read got -2
>> error during benchmark: -5
>> error 5: (5) Input/output error
>>
>> not sure what that's about...
>>
>
> Oops... I typo'd --no-cleanup. Now I get:
>
>     sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>       0       0         0         0         0         0         -         0
>   Total time run:        0.243709
> Total reads made:     1292
> Read size:            4096
> Bandwidth (MB/sec):    20.709
>
> Average Latency:       0.0118838
> Max latency:           0.031942
> Min latency:           0.001445
>
> So it finishes instantly without seeming to do much actual testing...

My bad.  I forgot to tell you to do a sync/flush on the OSDs after the 
write test.  All of those reads are probably coming from pagecache.  The 
good news is that this is demonstrating that reading 4k objects from 
pagecache isn't insanely bad on your setup (for larger sustained loads I 
see 4k object reads from pagecache hit up to around 100MB/s with 
multiple clients on my test nodes).

On your OSD nodes try:

sync
echo 3 > /proc/sys/vm/drop_caches

right before you run the read test.

Whatever issue you are facing is probably down at the filestore level or 
possible lower down yet.

How do your drives benchmark with something like fio doing random 4k 
writes?  Are your drives dedicated for ceph?  What filesystem?  Also 
what is the journal device you are using?

Mark

>
> James
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>