Re: Ceph write performance

[George Shuklin <shuklin@selectel.ru>]

On 20.07.2012 14:41, Dieter Kasper (KD) wrote:

Good day.

Thank you for attention.

ramdisk size ~70Gb (modprobe brd rd_size=70000000)
journal seems be on same device as storage
size of OSD was unchanged (... means I create it by manual and do not 
make any specific changes)

During test I watch IO load closely, IO on MDS/MON was insignificant 
(most of the time zero, sometimes few very mild peaks).

Just in case, configs:

ceph.conf:

[osd]
         osd journal size = 1000
         filestore xattr use omap = true

[mon.a]
         host = srv1
         mon addr = 192.168.0.1:6789

[osd.0]
         host = srv1

[mds.a]
         host = srv1

fio.ini:
[test]
blocksize=4k
filename=/media/test
size=16g
fallocate=posix
rw=randread
direct=1
buffered=0
ioengine=libaio
iodepth=32


Thanks for advising, I'll recheck with new settings.

> George,
>
> please share more details of your config:
> - RAM size of your system
> - location of the journal
> - size of your OSD
>
> Can you try (just for the 1st test) to
> .. put the journal on RAM disk	
> .. put the MDS on RAM disk
> .. put the MON on RAM disk
> .. use btrfs for OSD
>
> As an alternative to isolate the bottleneck you can try to
> - run without a journal
> - use RBD instead Ceph-FS
>    + create a File System on top of the /dev/rbd0
>
> Regards,
> Dieter Kasper
>
>
> On Fri, Jul 20, 2012 at 12:24:15PM +0200, George Shuklin wrote:
>> Good day.
>>
>> I've start to play with Ceph... And I found some kinda strange
>> performance issues. I'm not sure if this is due ceph limitation or my
>> bad setup.
>>
>> Setup:
>>
>> osd - xfs on ramdisk (only one osd)
>> mds - raid0 on 10 disks
>> mon - second raid0 on 10 disks
>>
>> I've mount ceph share at localhost and run FIO (randwrite, 4k, iodepth=32)
>>
>> What I've got: 1900 IOPS on writing (4k block, 1Gb span).
>>
>> Normally fio shows about 200kIOPS writing on ramdisk.
>>
>> Why it was so slow? I've  done setup exactly like described here:
>> http://ceph.com/docs/master/start/quick-start/#start-the-ceph-cluster
>> (but one osd).
>>
>> Thanks.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html