Re: Poor read performance in KVM

[Mark Nelson <mark.nelson@inktank.com>]

On 07/19/2012 01:06 PM, Calvin Morrow wrote:
> On Thu, Jul 19, 2012 at 9:52 AM, Tommi Virtanen<tv@inktank.com>  wrote:
>>
>> On Thu, Jul 19, 2012 at 5:19 AM, Vladimir Bashkirtsev
>> <vladimir@bashkirtsev.com>  wrote:
>>> Look like that osd.0 performs with low latency but osd.1 latency is way
>>> too
>>> high and on average it appears as 200ms. osd is backed by btrfs over
>>> LVM2.
>>> May be issue lie in backing fs selection? All four osds running similar
>>
>> Our typical experience with btrfs right now seems to be that it works
>> fast when the filesystem is fresh, but as it ages, it starts to have
>> higher and higher delays on syncing writes. This does not seem to be
>> completely deterministic, that is, if you run many btrfs'es, the
>> symptoms start to appear at different times on different instances.
>
> Is Inktank seeing the slowdown on btrfs volumes with large metadata
> (32k / 64k) node/leaf sizes as well, or only on default (4k) sizes?
>
> Vladmir,
>
> What node/leaf size are you using on your btrfs volume?

Hi Vladmir,

We are seeing degradation at 64k node/leaf sizes as well.  So far the 
degradation is most obvious with small writes.  it affects XFS as well, 
though not as severely.  We are vigorously looking into it. :)

Mark

>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Mark Nelson
Performance Engineer
Inktank