Re: poor OSD performance using kernel 3.4

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

It would be really nice if somebody from inktank can comment this whole
sitation.

Thanks!

Stefan

Am 29.05.2012 05:54, schrieb Alexandre DERUMIER:
>>> This happens with ext4 or btrfs too. 
> 
> maybe this is related to io scheduler ?
> 
> did you have compared cfq,deadline,noop scheduler ?
> 
> noop should be fast with ssd.
> 
> 
> also what's is your sas/sata controller  ?
> 
> ----- Mail original ----- 
> 
> De: "Stefan Priebe" <s.priebe@profihost.ag> 
> À: "Alexandre DERUMIER" <aderumier@odiso.com> 
> Cc: ceph-devel@vger.kernel.org, "Mark Nelson" <mark.nelson@inktank.com> 
> Envoyé: Lundi 28 Mai 2012 21:48:34 
> Objet: Re: poor OSD performance using kernel 3.4 
> 
> Am 28.05.2012 08:52, schrieb Alexandre DERUMIER: 
>>> I think filestore journal parallel works only with btrfs. 
>>> Other filesystem are writeahead. 
>>>> ... you might be right but i can't change ceph's implementation. 
>>
>> See my schema, 
>> I think you see parallel writes, because you see flush write of first wave to disk, in the same time 
>> of second wave write to journal. 
> Yes i fulllý understand and agree - but still this should at least 
> result in a constant bandwidth near max of underlying disk. 
> 
>>>> I totally aggree with you but this is just a test setup AND if you have 
>>>> a big log file to copy let's say 100GB your journal will never be big 
>>>> enough and the speed should never drop to 0MB/s. Also i see the correct 
>>>> behaviour with 3.0.X where the speed is maxed to the underlying device. 
>>>> So i still see no reason that with 3.4 the speed drops to 0MB/s and is 
>>>> mostly 10-20MB/s instead of 130MB/s. 
>>
>> Maybe something is wrong with 3.4, then your disk write more slowly. (xfs bug, sata driver controller bug, ...) 
> 
> This happens with ext4 or btrfs too. 
> 
> Squential write speed to FS is exactly the same under 3.0 and 3.4 using 
> oflag=direct. 
> 
> 3.4: 
> 10000+0 records in 
> 10000+0 records out 
> 10485760000 bytes (10 GB) copied, 41,4899 s, 253 MB/s 
> 
> 3.0: 
> 10000+0 records in 
> 10000+0 records out 
> 10485760000 bytes (10 GB) copied, 40,861 s, 257 MB/s 
> 
>> maybe some local benchmark of your ssd with 3.4 can give some tips ? 
> 
>>>> How many disks (7,2K) do you have by osd ? 
>>>>> One intel 520 SSD per OSD. 
>>
>> I see some benchmark on internet about 150-300MB/s (depend of the blocksize). 
> bench OSD shows around 260MB/s 
> 
> ceph osd tell X bench shows me a speed of 260MB/s under both kernels 
> which corresponds to the dd from above. 
> 
>> Something must be wrong, Doing local benchmark can really help I think. 
>> You can use sysbench-tools 
>> https://github.com/tsuna/sysbench-tools 
>> It make bench compare with nice graphs. 
> Thx hopefully i'll find something. 
> 
> Stefan 
> 
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html